Data - Import, Export, Backup

When referring to data in KDSmart we are referring to your Trials, Traits and Tags (TTT) which KDSmart manages in it’s own database located on your Android device.
To commence using KDSmart with your own Trials, that Trial data must first be loaded or imported into KDSmart.

There are a few things to consider beforehand. The following table will direct you to the necessary section(s) for guidance:

You need to ... Then ... (choose the appropriate link below)
1. Locate Your Files On The Device

Variations with appearance and behaviour of the interface on phones and tablets occurs due to changes between the different versions of Android.

This section provides some help to locate the files you have copied to the device, ready for import.

2. Load Or Import Data Into KDSmart To Start Scoring

Several methods of importing or loading data into KDSmart are:

  • Using CSV files (for using KDSmart ‘standalone’);
  • Connecting to KDXplore directly to Import; or
  • Directly connecting to a KDDart database.
3. Export Scored Data Scored, or partially scored, Trial data can be exported for review, analysis or curation as required using the same methods as used for import.
4. Backup KDSmart The KDSmart database on the device can be exported, such as for backup or transfered to another device.


What Data Transfer Method is Best?

This is dependent upon several factors influenced by your organisation’s implementation of KDDart platform components either fully, partially or not at all. For example, if KDSmart is being used ‘standalone’ without any other KDDart software, CSV files are required to load your data.

The following table outlines

Best When ... Description
KDSmart is to be used on it’s own

Using KDSmart ‘standalone’, without any other KDDart software, means CSV files must be used to import and export all your Trial, Traits and Tag data.

Also useful if you want to try KDSmart with some of your Trial data just to see how it works in the field.

Several KDSmart devices used to score the same trial and KDXplore is being used Using KDXplore with KDSmart when a direct connection can be established for data transfer, or if connection is not possible either CSV or KDX files can be used.
Your Trials are stored in KDDart’s data layer and a direct connection is available This functionality is returning to KDSmart to directly connect with the KDDart database layer for Trial selection and download or upload.


Locate Your Files On The Device

Some users have expressed difficulty finding their files for importing, e.g. located on the SD card , which may be hidden from view and hence file selection.

This section is relevant when importing data into KDSmart from either CSV or KDX files. When selecting Trials, Traits or Tag files for import, a different behaviour will occur depending upon the Android version in use.

Background:

Android 4.4, know as ‘KitKat’, saw the introduction of the Storage Access Framework or SAF which provided a new way to browse and open documents, images and other files on the tablet or phone.

The following table shows KDSmart Import/Export functions and whether the SAF or the original Android file chooser will be displayed for the particular Android version in use. Note: The centre Kitkat column shows the variation that will be visible between KDSmart functions.

KDSmart Functions/Android Versions using SAF
KDSmart Function Android IceCreamSandwich (4.0.3) to
Jellybean (4.3.1)
Android Kitkat
(4.4 to 4.4.4)
Android Lollipop +
(5.0 to + )
Trial CSV Import
FileChooser
SAF
SAF
Trial KDX Import
FileChooser
SAF
SAF
Trial CSV Plots Export
FileChooser
SAF
SAF
Trial CSV Plots + Sub-Plot Export
FileChooser
FileChooser
SAF
Trial Zip Export
FileChooser
SAF
SAF
Trial KDX Export
FileChooser
SAF
SAF
Multiple Trial Export Plots CSV
FileChooser
FileChooser
SAF
Multiple Trial Export Plots + Sub-Plot CSV
FileChooser
FileChooser
SAF
Multiple Trial Export Zip
FileChooser
FileChooser
SAF
Multiple Trial Export KDX
FileChooser
FileChooser
SAF
Trait CSV Import
FileChooser
FileChooser
SAF
Trait CSV Export (Multiple Traits included)
FileChooser
SAF
SAF
Tags CSV Import
FileChooser
FileChooser
SAF
Tags CSV Export (Multiple Tags included)
FileChooser
SAF
SAF
Database Export
FileChooser
FileChooser
SAF

The following animated illustrations show how choosing a file differs between three different Android versions.

Import Dialog Comparison using Different Android Versions Performing the Same Function
FileChooser
With JellyBean
SAF
With Kit Kat
SAF
With Lollypop

Import with JellyBean (select to zoom)

Import with Kit Kat (select to zoom)

Import with Lollypop (select to zoom)



Load Or Import Data Into KDSmart To Start Scoring

Several methods are available for loading or importing data (i.e. Trials, Traits and Tags) into KDSmart. These instructions apply to using Comma Separated Values (CSV) files.

KDSmart can operate ‘Standalone’ which means without any other companion KDDart software which relies upon using CSV files to transfer your Trial data.

A Trait or Tag, once defined in KDSmart, is usable across multiple Trials and Nurseries, hence importing all Traits or Tags is not an ‘every time’ requirement. When importing, any pre-existing Traits or Tags will not be overwritten.

It is important to note ‘common’ Traits also ‘share’ the validation rule. Should this be an undesirable behaviour, a separate similar named Trait would be required for each alternate Trait and validation.

Note

  • Before proceeding with an import the CSV files will need to be copied onto the device (tablet or phone).
  • The Traits to be referenced by a Trial must be imported into KDSmart before the Trial.
  • For resources: See the KDXplore/KDSmart Example Files page.
  • Before proceeding with an import the CSV or KDX files must be copied onto the device (tablet or phone). This help does not show how to copy files to/from the phone/tablet other than using KDXplore’s KDXchange.


First Steps for Trials, Traits and Tags

Dependency: To satisfy data dependency when importing Trials using CSV files the Traits referenced by a Trial Must be imported into KDSmart, or already exist, Before the Trial.

Once a Trait or Tag is defined in KDSmart it can be referenced by multiple Trials or Nurseries, hence importing is not required every time. When importing a Trial, any pre-existing Traits and Tags will not be overwritten.

It is important to note that Traits which are ‘common’, i.e. referenced by multiple Trials, also ‘share’ the Trait’s validation rule. If this is undesirable a separate, similar named Trait would be required.

Note: If required “helper” Excel templates for preparing data for use in KDS/KDX are available, along with further documentation at software.kddart.com/KDSmart

For Trials, Traits and Tags, the following Import steps are common, described in the following table:

Steps to Import Trials (part 1 of 2), Traits and Tags
Step Action
1 Select the import button down-action to display the following menu tt-imp-menu for Traits and Tags or trial-imp-menu for Trials:
2 Navigate to the appropriate directory on the device where the CSV file(s) are located and choose the required file. For a Trait or Tag the import will start when the file is selected.
3 Import Traits and Tags:
The import process is quick when a Trait or Tag CVS file is loaded and a brief message (as illustrated) displays the number of records imported. importtagmsg
Alternatively, if the records already exist in KDSmart an Import Failed message will display citing the reasons (i.e. Tag or Trait already exists).
On completion the Trait or Tag window will display with the new records added.
4 Import Trials: - Continue with the following Import Trials From CSV section….


Import Trials From CSV

These instructions continue from the previous section, First Steps for Trials, Traits and Tags.

Steps to Import Trials (part 2 of 2)
Step Action
5 Ensure the previous four steps have been performed and a CSV file has been selected for Import resulting in a similar Import window to the following ‘Import Trial from CSV’ illustration.
6 Make any column selections and adjustments using the following information as a guide, taking heed of any warnings.
Ensure the Trial Layout is specified as you require as this is determined at import - see the following notes.
7 Once finished column selections select the Import button. This button will become active when mandatory columns have been specified.

Note: Import functionality is essentially the same for both KDSmart and KDXplore as shown in the following illustrations. They show the commencement of import for the same CSV file with default values, before any adjustments.

CSV Trial Import Example Showing Interfaces For KDSmart and KDXplore
KDSmart Import Window
_images/kds-import-trialmapping.png

KDSmart Import Trial from CSV Window

KDXplore Import Window
_images/kdx-import-trialmapping.png

KDXplore Import Trial from CSV Window

When making selections for the import the Import Trail CSV screen, illustrated above, shows the following columns:

Import Screen Column Descriptions For KDSmart and KDXplore
Displayed Column KDSmart Displayed Column KDXplore The Column Number in the CSV file
CSV Heading Column Heading The Column Heading Name from the CSV row 0.
Data Row #1 First Row Data Data Row #1 shows the data contents at row 1 to assist with making selections.
Import As Import As Instructs KDSmart/KDXplore how to import this column of data from the CSV file. In other words classify if the data is a Plot, Trial attribute, Trait or ‘don’t import’.

It is very important to note and distinguish the following:

  • Plot Attributes have constraints, such as the Row, Column must be unique in the CSV;
  • PlotID - If chosen must be unique in the CSV;
  • PlotBlock - If chosen must be unique in the CSV;
  • Row, Column is required when PlotID or PlotBlock is NOT selected; and
  • TrialName is derived from the CSV file name, or if present the column chosen as TrialName.

After selecting a file to import each column in the CSV must be assigned an attribute type. This determines how the data in the column will be interpreted. Columns can be selected as NOT import as well as whether the data is related to the Trial, a Plot or a Trait Name to be scored. If the file contains values for Traits these will be loaded into KDSmart.

Key column names are automatically identified and paired with KDSmart columns, however this may be overridden by using the drop-down list next to each CSV Heading to direct how the data in the column is interpreted. Values from the first “data” row are displayed to assist you in this.

For each row of the display selection will display the Column Selection list as shown in the next illustration.

Note: The scrollbar disappears from view and it may not be obvious this is a scrollable list.

_images/kds-import-column-dd.png

Import Column Mapping
Drop Down

  • Key column names are automatically identified and paired with KDSmart columns.
  • Columns where there is not a match are assumed to be a Trait Name.
  • To record the same Trait on successive visits to the Trial site, you may use the Trait Instance functionality in KDSmart to do this; (this is not the same as the date/time scored value that KDSmart records with every measurement that you take).

Trait Instances provide a “temporal” separation to avoid the definition of separate Traits for each scoring activity. KDSmart provides three alternatives to choose from on how to interpret the tail portion of a Trait Name:

  • No Trait instances (no special interpretation);
  • A colon ‘:’ followed by the instance number; and
  • Two underscores followed by the instance number.

Traits are defined to have a Trait Level. This tells KDSmart whether the Trait is for recording measurements for a Plot or for a Sub-Plot within the Plot.

If the Trait has not been previously defined, you may choose to create the Trait as a ‘Trait for Sub-Plot’. If, however, the Trait is already defined, KDSmart will automatically use the current definition.

The names marked with one (*), two asterisks ( **) or three asterisks ( ***) must be provided before you can import. You MUST provide at least one of:

  • Case 1: ** Plot Column (X) AND Plot Row (Y) OR
  • Case 2: * Plot Id; OR
  • Case 3: ** Plot Column (X) AND ** Plot Row (Y) AND *** Plot Block.

Note

If only the Plot Id is provided you should later edit the Trial Layout (see case 2 below) to set the Origin Plot Id (this should be less than or equal to the lowest valued Plot Id) and the Run Length. This allows KDSmart to display the Trial’s physical layout in the Field View.

Sample Measurement data in CSV files will be imported and assigned a date/time scored of the time of the CSV import.


Three Trial Layout Use Cases

The following table describes the three main scenarios for Trial Layout. The Trial Layout Options shown will be visible after import when the Trial Details are viewed. It is then details such as Origin, Run Length, etc. can be set for the Trial.

Trial Layouts
Case Description Trial Layout Options
This Trial Layout represents an X and Y layout.
Each X,Y coordinate must be a unique pair within the Trial.
layoutcase1
This Trial Layout represents a Trial using PlotIDs which must be unique. layoutcase2
This Trial Layout represents a Trial using Blocks. Blocks must be unique and must contain a unique X,Y pair within a block. layoutcase3


Backup KDSmart

If required the KDSmart database can be exported then copied to another device, e.g. to memory card, to maintain as a backup. It can also be transfered to another Android device running KDSmart.

Steps to export the KDSmart Database
Step Action
1 At the Home screen or menu by select Manage Devices kds_devices
2 Within the Database tab select the folder to locate the database export file (see following illustration).
3 Select the Database Files option.
4 Select the Disk button to save the database.

KDSmart Database Export (select to zoom)