Timestamp;Year;Short/Full Paper;Please select your nominated AGILE Best Paper.;Have you considered the reproducibility of research published in your nominated paper?;Do you agree with our rating? Please comment;Please rate how strongly the following circumstances have hindered you from providing all data, methods and results used/developed during your research? [The need to invest more time into the publication];Please rate how strongly the following circumstances have hindered you from providing all data, methods and results used/developed during your research? [Lack of knowledge how to include data/methods/results into the publication];Please rate how strongly the following circumstances have hindered you from providing all data, methods and results used/developed during your research? [Lack of tools that would help to attach data/methods/results to the publication];Please rate how strongly the following circumstances have hindered you from providing all data, methods and results used/developed during your research? [Lack of motivation or incentive];Please rate how strongly the following circumstances have hindered you from providing all data, methods and results used/developed during your research? [Legal restrictions (e.g. privacy issues or copyright)];Please add here if there were any other hindering circumstances;What would you suggest to AGILE community to encourage publishing fully reproducible papers?;We would like to ask for your consent to discuss your answers to this survey in our publication.;
10/23/2017 23:47:43;2013;Short;"Schwering et al. (2013) ""Orientation Information in Different Forms of Route Instructions""";Yes, it is important to me that my research is fully reproducible;yes. It was a short paper mostly on ideas. Thus, data was lacking, algorithm was specified (but it did not yet exist in very detail);Moderately hindered;Moderately hindered;Not at all;Not at all;Not at all;;I would publish my data without incentive, if tools and procedure existed;Yes, I agree that my answers will be discussed in a publication.;
10/23/2017 23:52:49;2016;Full;"Almer et al. (2016) ""Critical Situation Monitoring at Large Scale Events from Airborne Video Based Crowd Dynamics Analysis Alexander""";Yes, I have somewhat considered reproducibility;Yes;Strongly hindered;Not at all;Not at all;Strongly hindered;Main reason;time, money, not possible for a company to share source code;no;Yes, I agree that my answers will be discussed in a publication.;
10/23/2017 23:55:57;2012;Full;"Magalhaes et al. (2012) ""A new method for computing the drainage network based on raising the level of an ocean surrounding the terrain""";Yes, it is important to me that my research is fully reproducible;I don't agree. Our input datasets are available at the NASA SRTM website. We also describe our computational environment (a regular desktop computer) and one could reproduce it easily (although the computer used in the experiments -- with a Core 2 Duo processor - is outdated nowadays).;Strongly hindered;Strongly hindered;Strongly hindered;Slightly hindered;Not at all;;I enjoy the Reproducibility Stamp initiative (http://www.replicabilitystamp.org/) and I think that it would be excellent if AGILE created a similar project.;Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 0:54:30;2013;Short;"Stein & Schlieder (2013) ""A Geowiki for Participatory Mobility 2 Volunteered Geographic Information""";Yes, I have somewhat considered reproducibility;I did not check the papers of the other authors. I agree with the rating ouf our paper, our concern was to clearly describe the algorithm.;Moderately hindered;Not at all;Not at all;Not at all;Not at all;From my point of view what is missing are the details of the simulation. To have a full description of the simulation in the paper would simply take too much space without giving that much value to the reader (our paper was a short paper). We could have put the source code for the simulation online somewhere. This would take additional work and time as we would have to clean up the program code, add comments, to make it readible for others.;"As papers have length restrictions (page count/word count) you need some platform where one can upload raw data, program code, etc. and which stays accessible for a long time. The data should be uploaded in a structured way which is usable for others (this can include e.g. huge databases of spatial data).
The community has to value the work of collecting and preprocessing the data as an important work on its own (otherwise one group has all the work of producing the data and other groups use it to write papers and get the reward).";Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 1:01:16;2010;Full;"Raubal & Winter (2010) ""A spatio-temporal model towards ad-hoc collaborative decision-making""";Yes, it is important to me that my research is fully reproducible;"""Not applicable"" is appropriate, because this paper is a conceptual paper. Specifications can be tested for consistency, but there is no reproducibility.";Not at all;Not at all;Not at all;Moderately hindered;Main reason;This particular paper had no data. But in principle, much of the data we are using is protected for privacy reasons - and this is a challenge for the whole community.;As far as ethical or legal issues of publishing the data are in the way there is no solution to the dilemma. If it is only lazyness / convenience that could be addressed by a policy of AGILE demanding publication of the data.;Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 5:27:17;2012;Full;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;
10/24/2017 5:29:05;2013;Full;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;
10/24/2017 12:46:37;2017;Short;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;
10/24/2017 13:42:05;2014;Full;"Scheider et al. (2014) ""Encoding and Querying Historic Map Content""";Yes, I have somewhat considered reproducibility;I agree. I would like to add that, the technologies underlying GIScience change very fast and the funding has time windows . Standards guarantee data availability even after the project is over. But code is different, specially when it depends on other technologies. Code is hard to maintain beyond the founding (and the research project) is over.;Slightly hindered;Not at all;Main reason;Strongly hindered;Not at all;Technical obsolescence;;Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 14:54:33;2014;Short;"Wiemann & Bernard (2014) ""Linking crowdsourced observations with INSPIRE""";No, I was not concerned with it;Just partly. The short paper is about introducing the concept and first ideas for linking different datasets. The focus was accordingly not on actual data, algorithms or results. Thus, I'd rather replace the zeros with x.;Main reason;Not at all;Not at all;Not at all;Not at all;;"I strongly support the idea of reproducable research for full paper publications. If applicable, it should be considered and rated in the review process, especially for best paper suggestions. However, I would relax this in the case of short papers, which are often used to present first ideas and start discussions. Here, full reproducibility requirements would put an unneccessary burden on authors. 

On the website, a section to promote reproducable research (links to data, code and results) could be established to enhance the reputation of the conference. Nevertheless, the major obstacle for reproducable research of AGILE papers is the limited access to papers (e.g. TU Dresden has no corresponding subscription)...but that's another issue, I guess.";Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 15:21:46;2016;Full;"Wiemann (2016) ""Spatial Data Relations as a Means to Enrich Species Observations from Crowdsourcing""";Yes, I have somewhat considered reproducibility;"Partly. Even though I documented the methodology and results, I'm curious, if anyone is really able to reproduce all of the results based on that documentation (I hope so, but I'm not fully confident). At that point, I'm surprised that ""available upon request"" counts as ""unavailable"". I'd differentiate here, because there might be good reasons for not putting everything online.";Main reason;Not at all;Not at all;Not at all;Not at all;The main reason for not putting R-Scripts online was the missing time (and to be honest missing motivation) for code cleanup;;Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 16:24:55;2017;Short;"Haumann et al. (2017) ""Energy-based Routing and Cruising Range Estimation for Electric Bicycles""";Yes, it is important to me that my research is fully reproducible;I think so. You'll find a much more detailed explanation of all preprocessing, algorithms and the computational environment in the master's thesis of Simon Haumann, upon which our contribution was based (and which should be referenced).;Moderately hindered;Slightly hindered;Slightly hindered;Moderately hindered;Slightly hindered;I think one often has the feeling that all data, code, etc. should be formatted and documented in a proper way - which can be a time-consuming task (before the submission deadline there is no time, and afterwards not really a strong incentive except making the research reproducible).;"Provide an easy way to attach data, code, etc. to the publication, and urge researchers to use this possibility (or make repositories ""open"", etc.).";Yes, I agree that my answers will be discussed in a publication.;
10/24/2017 18:22:42;2013;Short;"Schwering et al. (2013) ""Orientation Information in Different Forms of Route Instructions""";Yes, it is important to me that my research is fully reproducible;yes;Slightly hindered;Moderately hindered;Strongly hindered;Strongly hindered;Main reason;page /submission limit or requirement ;;Yes, I agree that my answers will be discussed in a publication.;
10/25/2017 8:44:00;2014;Full;"Groechenig et al. (2014) ""Estimating completeness of VGI datasets by analyzing community activity over time periods""";Yes, it is important to me that my research is fully reproducible;Since the used data is data from OpenStreetMap, I do not understand a rating of 2 concerning data. Concerning the algorithms, the analytics framework has been released as open source in the meanwhile: https://github.com/SGroe/vgi-analytics-framework;Main reason;Not at all;Not at all;Not at all;Moderately hindered;Time is in most cases the main reason. Opening a source code is always a time-consuming step. Selecting an appropriate license being compatible with other licenses of linked frameworks or source codes is also challenging.;Make it a basic requirement to get nominated for the best paper award;Yes, I agree that my answers will be discussed in a publication.;
10/25/2017 16:59:36;2014;Full;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;
10/26/2017 16:21:17;2016;Short;"Josselin et al. (2016) ""Sonorous Cartography for Sighted and Blind People""";No, I was not concerned with it;I agree about the data. I would say the method may be reproducible but to be honest, it is very difficult to find blind people ready to participate. It was very hard to convince them. About the results, we recently fund some publications that showed different results from us. (meaning that blind people do not have better compensating capabilities in hearing indeed).;Moderately hindered;Not at all;Moderately hindered;Strongly hindered;Not at all;;Very good idea. That might be a common research European project. As the next director of my lab (ESPACE in France), I would encourage my colleagues to participate.;Yes, I agree that my answers will be discussed in a publication.;
10/27/2017 11:09:20;2010;Full;"Raubal & Winter (2010) ""A spatio-temporal model towards ad-hoc collaborative decision-making""";Yes, I have somewhat considered reproducibility;Not completely. Although we only present a formal model, the formal examples in the paper are reproducible.;;;;;;You marked all categories as not applicable so these questions are also not applicable.;It's part of good scientific practice and becomes more and more important. Sometimes it's difficult though, especially when you work on research with sensitive (privacy!) data.;Yes, I agree that my answers will be discussed in a publication.;
10/27/2017 14:14:52;2016;Short;"Rosser et al. (2016) ""Full Meta Objects for flexible geoprocessing workflows: profiling WPS or BPMN?""";Yes, it is important to me that my research is fully reproducible;data is  3 I think to remember ;Not at all;Not at all;Not at all;Not at all;Not at all;I think the work is reproductible;A prize? A special award or refund of some of the fees ;Yes, I agree that my answers will be discussed in a publication.;
10/29/2017 13:46:52;2016;Short;"Rosser et al. (2016) ""Full Meta Objects for flexible geoprocessing workflows: profiling WPS or BPMN?""";Yes, it is important to me that my research is fully reproducible;No - I would have considered that the ideas presented in this paper and therefore the results are testable as the specific data set is not the key issue. The paper was about the relative merits of two open standards approaches to  the processing of metadata for quality assurance. ;Slightly hindered;Slightly hindered;Not at all;Not at all;Not at all;More a question of focus of the paper (exploring the pros and cons of two open standards approaches) and the return in terms of benefit (to the authors as well as the wider community) against the effort required to make available for the longer term any specific data sets used. The mere saving of data sets is often not sufficient unless associated with rich metadata regarding a range of parameters surrounding it collection, categorisation, how it was applied, etc.;As a research-led Europe-wide organisation AGILE is well placed to set out and promote good practise.  People are keen to present at AGILE Conferences and there is therefore the opportunity to leverage compliance to any good practise guidelines AGILE promotes. An AGILE hosted data and research themes repository  might be useful but the investment in time and money will be more than nominal and it would need to persist over a long period to deliver real value.. Local storage is often ineffective as research groups come and go - people move institution and retire and the ability to then access stored data - even if still in preserved - becomes problematic.   Importantly AGILE might encourage and promote enduring cross research institute and cross country collaboration.  The EU supports this through the FP xxxx and now  H2020 programmes but the inefficiency and poor value for money of these programmes  means a different approach needs to be found. Initiatives such as yours driven and shaped by active researchers are very valuable to  raise the importance of the issues and explore more efficient/effective ways of achieving reproducability and collaboration (all I suspect on a very modest budget?);Yes, I agree that my answers will be discussed in a publication.;
10/30/2017 10:18:43;2017;Full;"Knoth et al. (2017) ""3D Building Maps for Everyone - Mapping Buildings Using VGI""";It is part of a project and we can only share parts of the work.;;Not at all;Not at all;Slightly hindered;Not at all;Main reason;;;;
10/30/2017 13:53:08;2013;Short;"Schwering et al. (2013) ""Orientation Information in Different Forms of Route Instructions""";Yes, I have somewhat considered reproducibility;kindly specify what documented and recreatable in this context.;Strongly hindered;Moderately hindered;Strongly hindered;Strongly hindered;Strongly hindered;;evoke interest among researchers on reproducibility of research;Yes, I agree that my answers will be discussed in a publication.;
11/2/2017 11:13:35;2014;Full;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;
11/10/2017 15:40:51;2017;Full;"Mazimpaka & Timpf (2015) ""Exploring the Potential of Combining Taxi GPS and Flickr Data for Discovering Functional Regions""";Yes, it is important to me that my research is fully reproducible;I agree;Not at all;Strongly hindered;Main reason;Not at all;Slightly hindered;;open access to all papers including their own website space in which data and tools might be uploaded and explained;Yes, I agree that my answers will be discussed in a publication.;
11/20/2017 13:38:20;2015;Full;The author has not agreed that one's answers would be included in the publication.;;;;;;;;;;;