Importing Data Manually

Introduction

The Importing activity allows you to import data into the database with various file formats. This data can be for Channels, Site information, Meter Readings or Associated Data. The importing can be manually done as and when required (see below), or the importing can be scheduled to load at a desired frequency picking up data from an email account, FTP or FTPS.  To schedule imports, click here for more information. 

You can choose to import data immediately, or schedule it to regularly import automatically.

To schedule imports to run automatically on a recurring basis, click here.

This section will show you how to import data immediately.  

Importing data 

  • Click on Importing

      

  • Click 

Adding an Importer

If the Importer isn't in the Favourites list, you need to add it in.

  • Click  to open the search wizard

         


  • Enter the name of your data source (e.g. "IMServ" or "SIPe") and search to find an appropriate importing option (buffer), select it and click OK
  • Click Next to confirm your choice 
  • Click Finish
  • Click   
  • Click 

The Importer will now show in the Favourites list and ready to be used. 

Note: You will not need to do this the next time you import, as the Importer will already be in the Favourites list.

Importing the Data (once the Importer is in the Favourites list)

From the Importing Activity:

  • Click 
  • Highlight the Importer you want to use
  • Click   to move to the Importer Options

         


  • Amend the import options as necessary, such as the time zone or date range to load (generally, no change is required)
  • Click   to move to the Source Options

         

This is where you specify where the file for importing should be retrieved from.

The options are:

  1. Import from file - the file is stored on the local computer and can be uploaded via the screen
  2. Import via Email - the file is attached to an email in a mailbox
  3. Import via FTP - the file is located on an FTP server


To manually import the data:

  • Click and browse to find the file you want to import
  • Click   to move to  finish the import

         


Note: Once you have uploaded the file, the Upload file option changes to Change file (if required)

         


  • Click   and a prompt will show to say the import request has successfully queued
  • Click  to allow you to carry out another import

         



Viewing the importing progress

The file will show in the Queued And Running Imports area (depending how long the import takes, sometimes it doesn't show)


Refreshing the Run History

  • Click on the Refresh icon in the Run History section
  • The screen will refresh to show the latest import, and its status (success or failed)

          


Click here for more details on the Run History section, including viewing run details and downloading log files.


Scheduling Imports using Email or FTP

Sigma is able to pick up files to import from email mailboxes and is able to support both Encrypted and Non-Encrypted versions of POP3 connection.

The secure version is often known as "Secure POP3" or "POP3S". When selected, Sigma's connection to the email inbox will be encrypted using SSL/TLS  (Secure Sockets Layer/Transport Layer Security).

When setting up an email importer, you will need to indicate whether the Encrypted or Non-encrypted version of POP3 should be used.  You will also need to specify the Port Number to use.  These details can be provided by your IT Team, but please see table below for commonly available email providers.


Gmail EncryptedGmail Non Encrypted
Server: pop.gmail.comServer: pop.gmail.com
Port:  995 Port: 110 


Office 365 (including Outlook, Hotmail and Live) EncryptedOffice 365 (including Outlook, Hotmail and Live) Non Encrypted
Server:  outlook.office365.comServer:  outlook.office365.com
Port:  995  Port: 110


To use the email import option, tick 

Setup the configuration options as per the below:

  • Email Server -  enter the email server address
  • Username - enter the email address
  • Password - enter the password for the email address
  • Protocol - choose either POP3 Encrypted or POP3 Non Encrypted
  • Port -  enter the Port number, this will default to 995 (encrypted) or 110 (non-encrypted)
  • Sender Address (Optional): This tells the system to only look for emails that have been received from a specific sender email address. Note a partial email address could be entered (e.g. @teamenergy.com would only search for emails received with the domain @teamenergy.com)
    • This is important if you are have more than one EDI Supplier sending emails to the one email address or you are using you normal email address for importing 
  • Subject (Optional): This tells the system to only look for emails that have been received with a specific subject. Note a partial subject string could be entered and is case insensitive (e.g. EDI File would find any emails that contained the words "EDI FIle" in it, even if the actual subject was "June Edi File Resend").
    • This is important if you are have more than one EDI Supplier sending emails to the one email address or you are using you normal email address for importing 

It is recommended that you test the connection to the mailbox before the schedule is saved.  This provides comfort that the system is able to connect before it runs for the first time and will not fail because of incorrect configuration and will provide details of why the connect was unsuccessful.

  • Click 

         



  • Click   to move to the next step


Scheduling the Importing via FTP

FTP stands for File Transfer Protocol and is a standard method for transferring computer files over local networks and the internet. The FTP works by having a program where the data is being transferred from known as a Server and another program that connects to the server known as the Client that requests the file(s) to be transferred and copies the data over. There are several different programs that work as either Server, Client or both.
One of the benefits of FTP is that it allows access to the files to be restricted to only certain users rather than making these publicly accessible.

Sigma is able to support both FTP and FTPS. The main difference being the added layer of security for FTPS which ultimately ensures the secure transmission of files between the the remote server and Sigma. FTPS stands for "File Transfer Protocol SSL (Secure Sockets Locker)". The SSL reference relates to the security aspect - a cryptographic protocol that encrypts the data being transferred.

Sigma has a built in FTP Client that can connect to an FTP Server to import data automatically via a schedule. To set up the automated importing an FTP Server will need to be set up. There are a large number of FTP Server Software providers offering a range of different functionality, TEAM would recommend that you pick the most suitable option for your business in line with your IT policies.

Whichever FTP Server program is chosen this will need to be set up with access to the folder where the files that need to be loaded to Sigma are sent to and able to be accessed from the internet. A user name and password will also need to be created to access the data, this will be entered in to Sigma when setting up the schedule.
If you want to limit access to the FTP further most Server programs will allow access to be limited to a range of IP addresses. Please contact Team Support if you require a list of the IP addresses used by Sigma.

The key configuration details for FTP are:

  • Host name -  the host name of the FTP server whereby the files are stored that need to retrieved
  • Username - the username to be able to access the FTP server
  • Password - the password to be able to access the FTP server
  • Path - the directory path to where the files are stored and need to be picked up from the FTP server

 Contact your IT department for the above details

         

  • Ignore root files - Tick if you want to go directly to the folder indicated in the Path (ie ignore the root files)
  • Secure FTP? (FTPS) -  Tick if you use FTPS (File Transfer Protocol Secure)
  • Implicit Secure FTP? -  Tick both FTP? and Implicit Secure FTP? if  using Implicit.

Implicit FTPS is actually more strict than explicit FTPS when it comes to establishing a secure connection. In fact, the entire FTP session is encrypted, in contrast to flexibility you have when using explicit FTPS

FTP vs FTPS

When using FTP alone, the data is sent between the two servers sending and receiving the data unencrypted. FTPS is an extension to FTP, using it as its foundation and adds a layer of security. It is a commonly used and simple way to protect your data when transferring data files into Sigma. 

FTPS stands for "File Transfer Protocol SSL (Secure Sockets Locker)", the SSL reference related to the security aspect - a cryptographic protocol that encrypts the data being transferred (note, the term SSL is used interchangeably with TLS or Transport Layer Security).

There are two types of FTPS - Implicit and Explicit, and Sigma is able to support both of them.

Explicit FTPS 

This is the current standard and is supported by the majority of FTP servers since it is an approved, standard way of protecting data. it also provides the most flexibility.

First a connection is made to the server using the same mechanism as traditional FTP. Once a connection is made, a secure connection is then established.

Implicit FTPS

When using implicit FTPS, the entire session is encrypted as an SSL connection is immediately established via port 990 before login or file transfer can begin. If the server receiving the request fails to comply with the security request, the server immediately drops the connection.


  • Click 

If successful, you will get the following prompt:

If unsuccessful, you will get the following prompt:

(Contact your IT Department or provider of the FTP server for advice if the FTP connection could not be established).


  • Click  to continue setting up the Schedule

Note:  Setting this up will enable Sigma to check the FTP Folder periodically and download all files in the Folder.  Therefore, you will need to implement an archive process, otherwise all the files will be downloaded each time.