TSG Doc - User contributions [en]

Presentation

2018-06-05T08:34:24Z

A.datadien: remove test

{{Infobox software
| name = Presentation
| logo = Presentation_name.png
| logo size = 200px
| logo alt = Presentation
| screenshot =
| caption =
| developer = Neuro Behavioral Systems
| installed version = varies <ref>See [[Pre-installed software]]</ref>
| installed version date = 
| platform = Windows
| language count =
| website = [https://www.neurobs.com/ neurobs.com]
| resources =
{{Infobox tsg
| child = yes
| downloads = {{bulleted list
| [[media:Presd183060216.zip|Installer for Windows (32-bit, zip)]]
| [[media:Presd201120417_64.zip|Installer for Windows (64-bit, zip)]]
| [https://www.neurobs.com/menu_presentation/menu_download/package_player Package player]
}}
| header2 = Templates
| data2 = *[[media:Template2017.zip|Template 2014 (zip)]]
}}
}}

Presentation is a Windows based programming tool that allows experimenters to set up and program all sort of experiments. It is the recommended software for time-accurate experiments and therefore supported by the institutes’ (Social Sciences Faculty, the MPI and the Donders Centre for Cognitive Neuroimaging) technical groups. The technical groups offer (PhD) students a couple of pre-programmed experiments (i.e. templates), which can be adjusted to build up own experiments. In this way, the (PhD) student can efficiently program experiments fitting an technical optimal environment. This is what this course is all about. It aims at teaching programming skills, which are needed to modify the existing templates such that they meet your own demands. Because this can be quite complex, we start with short assignments, which address one basic and simple problem at a time. They will all contribute to the final assignment in which you will work on an existing template.
<br/>

== Course ==

Find the presentation workshop documentation here: [[Media:Programming with Presentation 2013.pdf]]

Find the presentation pre-reader here: [[Media:Presentation pcl pre-read 2013 .pdf]]

== License ==

To run presentation in the labs you need a software dongle. Reserve a dongle when you book a lab. If you need a dongle for research outside the university you could discuss time limits with [[Ronny Janssen]].

When you want to run in your office use the network license. Boot presentation, choose option '''"activations" -> "Network"'''

Fill in the following within the "'''Network License Settings"''': 

{| class="wikitable"
|-
! Enable Hostname:
| lic-pres.socsci.ru.nl
|-
! Port:
| 1245
|-
! Timeout (ms):
| 5000
|-
|}

[[File:PresentationLicentie.jpg | 700px ]]

== Video ==
Presentation only supports non-proprietary video formats.<ref>https://www.neurobs.com/menu_support/menu_forums/view_thread?id=8785#i6</ref> Neurobs recommends encoding your videos using h264 or XVID, and using the AVI format.<ref>http://www.neurobs.com/wiki/Presentation/Video</ref> The TSG discourages the use of XVID and AVI, because these standards have become obsolete in recent years, but since we have had little success getting alternative non-proprietary video formats to work in Presentation,<ref>[[Video_Codecs#Software_support]]</ref> it seems the use of AVI is inevitable. You can find more information on how to export and convert your videos for Neurobs Presentation on our [[Video Codecs#Neurobs Presentation|Video Codecs page]].

== Troubleshooting ==

==== Version History ====

For version history please click here: http://www.neurobs.com/menu_presentation/menu_download/version_history

==== Fixed Bugs ====

For fixed bug / known bugs please click here: http://www.neurobs.com/menu_support/menu_help_resources/known_bugs

==== Forum ====

For various information please click here: http://www.neurobs.com/menu_support/menu_forums/contents_page

==== Wiki ====
For the wiki click here: http://www.neurobs.com/menu_support/nbs_wiki

==== Compatibility Issues ====

*Always check beforehand in which version the script was written.
*New versions do not always maintain backwards compatibility.
*Features can get removed, or changed.

==References==
<references/>

Presentation

2018-06-05T08:29:44Z

A.datadien: test download link

Main Page

2017-03-30T08:40:31Z

A.datadien: Add link to data files article

<div style="text-align:center; padding:0.2em; border:2px solid #56ad5a; margin:0.5em 0; font-weight:bold;">
If you are, or have been a customer of the Technical Support Group, please consider filling in our [https://survey.socsci.ru.nl/index.php/622997/lang-en TSG Customer Survey!]
</div>

Welcome to the TSG Wiki.

On this wiki you will find documentation for the Hardware, Software and Research Equipment used in the lab environments of the research facilities of the Faculty of Social Sciences, as well as other useful information.
To learn more about the Technical Support Group, visit [http://www.ru.nl/socialsciences/technicalsupportgroup/ our website] or the [[TSG info|about]] section.

<div style="width:100%;">

{{Main Page/Frame
| color = be311a
| title = Hardware Documentation
| content = {{Bulleted list
| [[Balance Board]]
| [[Behringer X-Air XR18]]
| [[Biopac]]
| [[Brainvision]]
| [[ButtonBoxes]]
| [[Computers]]
| [[Eyetrackers]]
| [[Head Mounted Displays]]
| [[JoySticks]]
| [[Kinect]]
| [[Laptops]]
| [[Monitors]]
| [[Optotrak]]
| [[Surveillance Camera]]
| [[TemperatureHumidity]]
| [[TMSi]]
}}
}}

{{Main Page/Frame
| color = be311a
| title = Software Documentation
| content = {{Bulleted list
| [http://docs.expyriment.org/index.html Expyriment]
| [[LimeSurvey]]
| [[Matlab]]
| [[Noldus Software]]
| [[Presentation]]
| [[Psychopy]]
| [[Python]]
| [[System Image]]
}}<br/>
;Miscellaneous
{{Bulleted list
| [[Workshop: Python for behavioural scientists]]
| [https://wiki.science.ru.nl/DCC/Main_Page DCC Cluster]
| [[Gitlab Social Sciences]]
| [[Virtualization Server]]
| [[Personal and Group drive Data storage ]]
| [[Video Codecs]]
| [[Data Files]]
}}
}}

{{Main Page/Frame
| color = be311a
| title = Lab Documentation
| content = {{Bulleted list
| [[BalanceBoard Lab]]
| [[Cubicles]]
| [[EEG Lab]]
| [[RIVER Labs]]
| [http://docs.sensorimotorlab.com SensoriMotorLab]
| [[Sound Recording Lab]]
| [[Tobii Eye Tracker]]
| [[VoiceKey Lab]]
}}<br/>
;Lab Use Policy and Guidelines
{{Bulleted list
| [[Booking labs]]
| [[Media:EEG_lab_user_guide_December2016.pdf|EEG lab user guide]]
| [[Media:Cleaningprocedure_EEG_September2016-1.pdf|EEG cleaning protocol]]
| [[Using labs]]
}}
}}

</div>

Data Files

2017-02-20T14:04:38Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number. This makes it ASCII compatible.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line. This simplifies stream reading since all records (lines) are terminated. This allows for the use of a readline() function for acquiring a line.
* The first line contains a header with column/field names.

===== Fields =====
* Fields are separated by the '''tab''' field delimiter, because they rarely occur in texts. This allows for the use of comma's and semicolons in sentences without using an escape character.
* The field delimiter should '''not''' be added after each line's last field. This allows for the use of a split() function for parsing a line.
* The last field in a line must not be empty, because it will show to parsers that the previous rule was obeyed.
* Fields are never surrounded by a quoting character.
* White space before or after field delimiters are considered part of a field.
* There is no defined escape character. If your data can contain tabs or newlines, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file encoding|| '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T14:01:20Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number. This makes it ASCII compatible.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line. This simplifies stream reading since all records (lines) are terminated. This allows for the use of a readline() function for acquiring a line.
* The first line contains a header with column/field names.

===== Fields =====
* Fields are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore escape characters are often not needed.
* The field delimiter should '''not''' be added after each line's last field. This allows for the use of a split() function for parsing a line.
* The last field in a line must not be empty, because it will show to parsers that the previous rule was obeyed.
* Fields are never surrounded by a quoting character.
* White space before or after field delimiters are considered part of a field.
* There is no defined escape character. If your data can contain tabs or newlines, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file encoding|| '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T14:00:42Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number. This makes it ASCII compatible.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line. This simplifies stream reading since all records (lines) are terminated. This allows for the use of a readline() function for acquiring a line.
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore escape characters are often not needed.
* The field delimiter should '''not''' be added after each line's last field. This allows for the use of a split() function for parsing a line.
* The last field in a line must not be empty, because it will show to parsers that the previous rule was obeyed.
* Fields are never surrounded by a quoting character.
* White space before or after field delimiters are considered part of a field.
* There is no defined escape character. If your data can contain tabs or newlines, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file encoding|| '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T13:40:59Z

A.datadien: changed word extension -> encoding

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number. This makes it ASCII compatible.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line. This simplifies stream reading since all records (lines) are terminated. This allows for the use of a readline() function for acquiring a line.
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should '''not''' be added after each line's last field. This allows for the use of a split() function for parsing a line.
* The last field in a line must not be empty, because it will show to parsers that the previous rule was obeyed.
* Fields are never surrounded by a quoting character.
* White space before or after field delimiters are considered part of a field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file encoding|| '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T11:23:21Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number. This makes it ASCII compatible.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T11:00:37Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

An example of what a file in this format may look like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T10:59:26Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== Example ==

This is what the file format looks like:
<pre>
User ID	Hair color	Response time	
1	brown	1.4	
2	blond	1230.434	
3	brown	0.399	

</pre>
An example file can be downloaded here [[File:Example.zip|thumb]] (sorry, it is zipped).

== Parsing ==
Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

== Alternatives ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

Data Files

2017-02-20T10:49:43Z

A.datadien:

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file is plain text for easy inspection and manipulation. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== File Format ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

== Parsing ==
Here is an example [[File:Example.zip|thumb]] file. Sorry, it is zipped. Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

Data Files

2017-02-20T10:43:48Z

A.datadien: Almost finished general specification, needs some explanations

== TSG suggested file format for experiment data ==
The TSG suggests a common file format for storing experimental data. Adhering to this format whenever practical makes it easier to re-use files and tools. The file format is a tab-separated values (tsv) file with the following specifications:

===== File =====
* File encoding is ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== Data =====
* For numbers the decimal separator is a dot, not a comma. There is no thousands separator.

== File Format ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

== Parsing ==
Here is an example [[File:Example.zip|thumb]] file. Sorry, it is zipped. Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

Data Files

2017-02-20T09:29:21Z

A.datadien: in progress save

== TSG suggested file format for experiment data ==
The TSG suggests the following file format for data storage for most experiments.

===== Lines =====
* Lines are separated by the '''\r\n''' line delimiter for better compatibility between operating systems.
* The line delimiter should also be added after the last line, because...
* The first line contains a header with column/field names.

===== Fields =====
* Field are separated by the '''tab''' field delimiter, because they rarely occur in texts and therefore require no escaping.
* The field delimiter should also be added after each line's last field, because...
* The last field in a line must not be empty, because... if there is no value, wat do...
* Fields are not surrounded by a quoting character.
* White space between field delimiters are considered part of the field.
* There is no defined escape character. If your data can contain tabs, use a different field delimiter or file format.

===== File =====
* The file is encoded as ASCII or UTF-8.
* The file contains no byte order mark (BOM) or other magic number.

* Last field must not be empty (what if no data?)

Explanation for these choices (also nice for ourselves). In progress..

#

== File Format ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

== Parsing ==
Here is an example [[File:Example.zip|thumb]] file. Sorry, it is zipped. Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>

Data Files

2017-02-16T16:17:20Z

A.datadien: start of simpler text (WIP)

== TSG suggested File Format For Dummies (WIP by Arvind for extra readability!) ==
The TSG suggests the following file format for data storage for most experiments.

* Field delimiter: tab (also after last field)
* Line delimiter: \r\n (also after last line)
* Quoting character: none
* File extension: "tsv"
* File encoding: ASCII / UTF-8
* Magic number: none
* First line contains header, not data
* Last field must not be empty (what if no data?)

Explanation for these choices (also nice for ourselves). In progress..

== File Format ==
Data can be saved in a lot of file formats. If there is no reason to do otherwise, we prefer delimited files with the options shown in bold. Alternative options are also shown.
{| class="wikitable"
|-
| file extension || '''tsv''' || csv || '''dat''' || txt
|-
| file extension || '''ascii''' || '''UTF-8''' || UTF-16BE || UTF-16LE || UCS-4/UTF-32
|-
| magic number || '''None''' || <BOM>
|-
| line delimiter || \n || \r || '''\r\n'''
|-
| line delimiter after last line || no || '''yes'''
|-
| field delimiter || '''<tab>''' || , || ;
|-
| field delimiter after last field || '''no''' || yes
|-
| quoting character || '''None''' || " || '
|-
| escape qc by doubling || no || yes
|-
| escape character || '''none''' || \
|-
| first line || '''contains header''' || contains data
|-
| last field in line || '''must not be empty''' || may be empty
|-
| whitespace following delimiter || '''part of field''' || not part of field
|-
| decimal separator || '''.''' || ,
|-
| thousands separator || '''none''' || . || ␣ || U+2009
|}
Note that tab characters and newlines cannot be present in field content.

== Parsing ==
Here is an example [[File:Example.zip|thumb]] file. Sorry, it is zipped. Importing such files can be done in many languages:
=== Python Standard Library===
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
for row in reader:
print(', '.join(row))
</nowiki>
or with header extraction
<nowiki>
import csv
with open('example.tsv', 'rb') as csvfile:
reader = csv.DictReader(csvfile, delimiter='\t', quoting=csv.QUOTE_NONE)
print(', '.join(reader.fieldnames)) # print header
for row in reader:
print(', '.join([row[key] for key in reader.fieldnames]))
</nowiki>
Note that when using Python 2 the field content will remain UTF-8 encoded (type=str). In Python3 strings will unicode (type=string).

=== Python Pandas ===
Pandas can interpret column type. You will have to store it separately or hardcode it.
<nowiki>
import pandas as pd

d = pd.read_csv('example.tsv', delimiter='\t', skip_blank_lines=False, quoting=csv.QUOTE_NONE)
</nowiki>
=== GNU R ===
<nowiki>
d <- read.csv("example.tsv", head=TRUE, sep = "\t")
</nowiki>