Skip to content

The list of research best practices

Below we list our catalog of 19 factual questions and their respective best practices. This document can be used to guide the process of gathering information about an artifact.

1) What?

What is it all about?

 1. The artifact shall have a name for reference
 2. The artifact shall indicate its context of development (e.g., domain, problem, project)
 3. The artifact shall indicate the main functionalities supported (e.g., language support, code generation, model analysis)
 4. The artifact shall indicate the relation with its respective paper

What does it have?

 5. The artifact shall provide an overall description of its directory structure and content
 6. The artifact shall include everything required for replications (i.e., complete)
 7. The artifact shall include no more assets than necessary for replications (i.e., concise)
 8. The artifact shall include a preprint of its associated article

What concepts and technologies underpin the artifact?

 9. The artifact shall indicate the theories that underpin its creation (e.g., formalisms, semantics)
 10. The artifact shall indicate the modeling languages used to develop it (e.g., UML, xtUML, SysML, BPMN)
 11. The artifact shall indicate the meta-modeling languages used to develop it (e.g., EMOF, CMOF, ECore/EMF, KM3)
 12. The artifact shall indicate the standards and/or specifications used to develop it (e.g., ISO, CMI, XMI, CWM, HUTN, JMI, DD, OCL)
 13. The artifact shall indicate the programming/markup language used to develop it
 14. The artifact shall indicate the libraries, dependencies, and frameworks used to develop it and their respective versions (e.g., Eclipse release)

2) Why?

Why it was created?

 15. The artifact shall report the motivation for its development
 16. The artifact shall report its objective/goal (e.g., verify claims, replicability, reusability, a whole new library/framework)
 17. The artifact shall indicate its advantages and/or novelty (e.g., algorithm, language, method)

3) Where?

Where is it hosted?

 18. The artifact shall be hosted in an open, public repository (e.g., GitHub, BitBucket, Zenodo, Figshare)
 19. The artifact shall be hosted in a repository indexed and findable by search engines (e.g., GitHub, BitBucket, Zenodo, Figshare)
 20. The artifact shall be archived for long-term, permanent access (e.g., Zenodo, Figshare)

Where shall I cite?

 21. The artifact shall provide an explicit format for citation (e.g., in a CITATION or README file)
 22. The artifact shall provide citation information in a BibTeX format
 23. The artifact shall provide an URL for citation
 24. The artifact shall provide a DOI for citation

 25. The artifact shall give credit to data obtained from other sources (e.g., authors, paper, repository, benchmark)
 26. The artifact shall provide references about key concepts (e.g., papers, surveys, wiki, reports)
 27. The artifact shall provide references to studies using it (e.g., known uses, integrated with)
 28. The artifact shall provide references to related artifacts/projects
 29. The artifact shall provide references using in-code citation (e.g., code header)

4) Who?

Who could use it?

 30. The artifact shall be deposited under an explicit open license (e.g., reported in a LICENSE file)
 31. Files shall be made available using open/non-proprietary formats (e.g., JSON, open XML schema)
 32. The artifact shall indicate a list of potential users (e.g., professionals, researchers, industry sectors)
 33. The artifact shall rely on open, well-maintained, and documented libraries or dependencies

Who are the authors?

 34. The artifact shall provide a communication channel for interacting with the community (e.g., forum, mailing list, issue tracker, IRC, Slack, Discord)
 35. The artifact shall indicate the names of its authors
 36. The artifact shall indicate the institution of its authors
 37. The artifact shall indicate the contact details of its authors/collaborators (e.g., email address, ResearchGate, Linkedin, website)
 38. The artifact shall indicate the ORCID of its authors/collaborators
 39. The artifact shall indicate the level of experience of its authors/collaborators (e.g., bio, degree, position)

Who funded this project?

 40. The artifact shall indicate the funding agencies that supported the project (e.g., ERC, NWO, CNPq, DFG, EPSRC, NSF)

5) When?

When did changes happen?

 41. Changes to the artifact shall be tracked using version control (e.g., GitHub, GitLab, BitBucket)
 42. Changes to the artifact shall be small (e.g., conciseness, cohesion, clear edit)
 43. Changes to the artifact shall be explained (e.g., CHANGELOG, commit messages)
 44. The artifact shall allow referencing or retrieving specific versions using tags and/or release identifiers

When do future changes shall happen?

 45. The artifact shall provide a timeline for future goals and planned updates (e.g., frequency, next steps, future work plans)
 46. The artifact shall be open for change requests and receiving feedback from users (e.g., bug fixes, pull requests, collaboration)

6) How?

How is it organized?

 47. Tabular data files shall follow analysis-friendly formats (e.g., the column is variable, the row is observation, data dictionary, the meaning of column/row headers)
 48. Files and folders shall have self-explaining names matching content, meaning, and human abstractions (e.g., doc/, src/, results/, src/, bin/)
 49. The artifact shall indicate best practices used (e.g., naming or code conventions, guidelines/checklists)
 50. Useful metadata shall be used as part of filenames for pattern matching (e.g., yyyymmdd), where meaningful
 51. The artifact shall be compliant with ICT accessibility standards (e.g., Section 508, WAI)
 52. The experiment workflow shall be broken-down into small and simple procedures to facilitate reuse (e.g., scripts, functions)
 53. The artifact shall include a website or wiki-page

How to set up a running environment?

 54. The artifact shall provide instructions for downloading
 55. The artifact shall provide the open-source code
 56. The artifact shall provide a binary/compiled version
 57. The artifact shall provide a container for freezing dependencies and quickly setting up a running environment (e.g., VM, Docker)
 58. The artifact shall provide a step-by-step tutorial of how to build the source code
 59. The artifact compilation shall rely on build automation tools (e.g., make, ant)
 60. The artifact compilation shall rely on dependency management tools (e.g., maven, pip)
 61. The artifact shall provide instructions to install it

How to get started?

 62. The artifact shall include instructions for running it on minimal test data (e.g., quick run, smoke testing)
 63. The artifact shall indicate the most relevant and interesting parts of the source code/artifact
 64. The artifact shall include step-by-step instructions for running it (e.g., README, PDF)
 65. The artifact shall include a video tutorial with the step-by-step for running it (e.g., Youtube, Vimeo)

How to replicate the experiment?

 66. The artifact shall include the complete set of test models analyzed
 67. The artifact shall provide instructions for manual/automated pre-processing of raw data for experiments (e.g., bash, python, R script)
 68. The artifact shall provide instructions for manual/automated replication of the complete (or at least a subset) experiment as in the paper (e.g., bash, python, R script)
 69. The experiment workflow shall be fully automated (including raw data processing, experiment execution, figures plotting)
 70. The artifact shall include the experiment results shown in its associated paper in tabular and machine-readable format (e.g., .csv, .tab)
 71. The artifact shall include log files produced

How to run the analysis of results?

 72. The artifact shall include scripts for the automated analysis of results as in the paper (e.g., R scripts, python scripts, Jupyter notebooks)
 73. The artifact shall include scripts for drawing figures and/or plots as in the paper (e.g., R scripts, python scripts, Jupyter notebooks)
 74. The artifact shall provide a clear description of the measurement procedures and metrics used in the paper

How could it be repurposed?

 75. The artifact shall indicate suggestions for contributions (e.g., notes.txt, todo.txt, ways it could be repurposed, wishlist)
 76. The artifact documentation shall be designed considering users with minimal expertise
 77. The artifact shall provide details about ethical concerns in replications
 78. The artifact source code shall be documented (e.g., in-code comments, Javadoc)
 79. The artifact shall provide instructions for integration/chaining with commercial tools (e.g., Matlab, DOORS)
 80. The artifact shall report known issues, bugs, and limitations (e.g., issue tracker)

7) How much/many?

How many resources does it need?

 81. The artifact shall indicate the system and environment where it was successfully evaluated (e.g., OS, CPU, RAM, GPU, Disk)
 82. The artifact shall indicate the minimum system and environment requirements for usage (e.g., OS, CPU, RAM, GPU, Disk)
 83. The artifact shall indicate the skills and/or settings required for usage (e.g., team configuration, users' skills)
 84. The artifact shall indicate the approximate amount of time needed to replicate the experiment