VegSpec: Full Testing Plan

PURPOSE:

The purpose of testing is to confirm that VegSpec provides relevant features for identifying and selecting species based on the site conditions, land use, and conservation practice, while also generating seed mixes and per-pound seed costs for the audiences of NRCS staff, Bureau of Land Management (BLM), Park Services, other partners, and Cooperative Extension agents.

Audience Descriptions

Experienced NRCS Staff

• Subject Matter Expertise: Highly experienced in natural resource conservation, agricultural practices, and regional ecosystems.

• Technology Experience: Varying levels of experience with computers and technology, potentially ranging from basic to proficient.

• Location: Typically serving a specific region or area for an extended period, with a deep understanding of local conditions and challenges.

New and Untrained Field Staff

• Subject Matter Expertise: May have limited experience in natural resource conservation or agricultural practices, especially if new to the role.

• Technology Experience: Varying levels of experience with computers and technology, potentially ranging from basic to proficient.

• Location: May be new to a specific region or area, requiring a learning curve to understand local conditions and challenges.

Partners

• Bureau of Land Management, Park Services, other partners: Main difference from NRCS is how land units are defined. Possibly focused on re-vegetation or ecological restoration (rather than conservation), especially after a fire. Might be using VegSpec for targeted seed collections.

  • Operational center staff: Higher experience

  • Field staff: Focused on re-vegetation, land treatment, limited application of a conservation practice
    
    

We expect these audiences to have different experiences in the following area:

1. Experience with subject matter (low to high)

2. Experience with technology / similar tools (low to high)

3. Location / area within which the individual provides services

4. Job function

5. Conservation Foci

Testing Aims

Through testing, we will be able to:

1. Identify any missing features crucial for successful decision-making,

2. Collect feedback on the overall effectiveness and helpfulness of VegSpec features for regional recommendations,

3. Check overall usability needs and concerns,

4. Assess key utilization differences across various audience backgrounds, and

5. Learn more about how the tool can live within an ecosystem of planning, implementation, and final report processes.

Hypotheses to Test:

Usage of Data:

Answers including your name, state, job responsibility, and perspective on the tool provided via a discussion and completion of a survey will be used to inform tool improvements. These responses will be shared with the development team at Precision Sustainability Agriculture. Responses will not be anonymous so that developers may contact respondents for clarification if needed.

Roles:

Facilitation Coordinator: Edit materials, support materials finalization, run mock testing sessions as needed, assist with tutorial creation, distribute testing materials, communicate with facilitators, remind them of upcoming deadlines, and answer questions → Elizabeth

Testing Lead: Draft materials, design the overall testing approach and strategy, run mock testing sessions as needed, and prepare the training approach → GameTheory

Facilitator: A trained individual who showcases the tool for respondents, asks them questions, and records their input at a testing session

Co-facilitator (optional): This individual joins a testing session with a facilitator to support the facilitator by taking notes during the session to allow the facilitator to focus on the group conversation.

Respondent: An individual who represents a user and speaks to their perspectives and needs and to how the tool would impact users. In testing sessions, they share their feedback with facilitators.

Timeline → https://precision-sustainable-ag.atlassian.net/wiki/x/AYBNKg

Target Participation:

• Total Respondents: 60

• Total Facilitators: 35

Each facilitator should test with 2-3 respondents:

• 1 high experience Conservation or field agent

• 1 limited experience Conservation or field agent

• [Optional] 1 partner from BLM, Park Services, field staff

Feedback is due by Friday, February 14th, 2025 at 4pm EST.

TESTING APPROACH:

The broad schedule is visible on GitHub.

PART I: Schedule Kickoffs: Schedule and book training sessions for facilitators. (Facilitation Coordinator)
PART II: Kickoff & Training: During the training sessions, walk facilitators through the tool, the materials, the testing approach, the timeline, and expectations. Offer tips and tricks and answer questions. (Facilitation Coordinator and Testing Lead)
PART III: Recruit Respondents: Review the materials and book testing sessions with respondents, preferably one-on-one. (Facilitators)

PART IV: Execute Sessions: Run testing sessions with respondents. Discuss the purpose of testing, have respondents use the tool, and ask the respondent questions. Record responses in the Facilitator Form. At the end of the session, have respondents fill out the VegSpec Respondent Survey. (Facilitators)

PART V: Analyze Results: Review results, tracking progress toward the testing goals. Review, organize, and present the results. (Testing Leads)
PART VI: Identify Insights and Next Steps: Share a slideshow of insights and a backlog of requests from the findings with the teams at PSA and NRCS. (Testing Leads)

SESSION FORMAT

Materials

For Facilitators

• Overview Doc that includes…

  • Timeline and targets

  • Session format

  • Links to Facilitator Form and Respondent Survey

  • Suggested recruitment email

• Facilitator Form: A document where they log info about their sessions and responses from respondents

For Respondents

• Respondent Survey: A document for recording responses into during the session

Session Format (one-on-one, 1 hour long):

Facilitators should request that respondents use a computer (not a tablet or phone) to partake in an in-person session or join a remote session. It is strongly recommended that sessions be one-on-one, not in groups.

1. Introductions & Overview (5 mins): Introduce yourself, ask respondents to introduce themselves (name, location, job title), and record their name(s) in the form. Give an overview of the session, explain why feedback is valuable, and explain how we’ll use the data. Ask for consent to continue.

2. Respondents Open VegSpec (5 mins): Share the VegSpec link with participants, ask them to open it, and ask them to share their screen so you may see what they’re doing. 

3. Questions about Pages (40 minutes): Following the script in the VegSpec Testing Facilitator Form, have respondents use the tool page by page and ask them the questions on the form. Also observe respondents’ reactions and record them on the form.

4. Discuss General Takeaways (5 minutes): Stop screen sharing and ask respondents the final questions.

5. Complete Surveys (5 minutes): Before ending the session, ask respondents to complete the Vegspec Respondent Survey. Meanwhile, review your notes for accuracy and complete the last section of the VegSpec Testing Facilitator Form. Once everyone submits their response, thank the respondents for their time and close the session.

Recommended Setup:

• We recommend all respondents and facilitators test over Zoom or another video conferencing setup. However, facilitators may also test in person.

• We recommend all respondents and facilitators have access to a computer or laptop so that they can see the tool and complete the form and survey.

• We do not recommend that VegSpec be tested on phones or tablets.

Respondent Survey Questions

About You

1. What’s your name?

2. What’s your email address (optional) – for developer use.

3. What organization do you work for? What’s your job title?

4. How long have you been working in this or a related role?

5. What’s your state?

6. What are the top three conservation practices you are familiar with or interested in?

7. How comfortable are you with technology? (5-point scale: “not at all comfortable” to “very comfortable”

Overall

1. How intuitive is the current version of VegSpec, upon first use?

   1. It is very intuitive and much easier to use than my current species selection tool.

   2. It is somewhat intuitive and slightly easier to use than my current species selection tool.

   3. It offers essentially the same user experience as my current species selection tool.

   4. It is somewhat confusing and slightly harder to use than my current species selection tool.

   5. It is very confusing and much harder to use than my current species selection tool.

2. How do you prefer to select an area of vegetative establishment in VegSpec?

   1. Draw your own polygon in the tool.

   2. Import a shapefile from Conservation Desktop, a Web Soil Survey, or other software.

   3. Use Well-known Text (WKT, a text markup language for representing vector geometry objects).

   4. I’m not sure.

   5. Other____________________

   6. If other, how?

3. How often do you typically have soil test results prior to planning a vegetative practice (most commonly for pH, phosphorus, and potassium)?

   1. Most of the time--this helps me plan fertilizer recommendations or better select plant species (for instance considering the pH value when planning a Conservation Practice Standard 550 range planting).

   2. Some of the time, but it really depends on the producer and the practice(s) involved.

   3. Seldom, the soil test is not a factor in species selection or applicability.

4. What is your most common planning scenario?

   1. I meet with the customer and we create specific seed mixes based on their objectives and site conditions.

   2. I prepare specific seed mixes based on the site conditions and my experience with plant materials; customers prefer my recommendations and have few opinions about species selection.

   3. Customers instruct me to work with their agronomist or wildlife biologist to create specific seed mixes based on their objectives and site conditions.

   4. Customers select commercially available seed mixes from their preferred vendor and request their use if they meet our specifications. 

5. When selecting plant materials, customers routinely select from the following characteristics (check all that apply):

   1. Active growth period

   2. After-harvest growth rate

   3. Anaerobic tolerance

   4. Bloom period

   5. Drought tolerance

   6. Growth form

   7. Growth rate

   8. Height at maturity (feet)

   9. Root depth, minimum (inches)

   10. Moisture use

   11. Nitrogen fixation

   12. Salinity tolerance

   13. Shade tolerance

   14. Toxicity

Value
Scale: 1= Strongly Disagree, 5= Strongly Agree

• VegSpec has functionality I value; I would make use of it in my day-to-day work.

• The tool asks for information in a way that is intuitive and logical.

• I understood where the data was coming from in the tool.

• I think VegSpec would help me make accurate and informed decisions.

Usability

• I found VegSpec easy to navigate, I could find what I needed without getting lost or confused.

• Terminology in VegSpec was easy to understand and clearly labelled.

• I feel confident I could use VegSpec efficiently without encountering significant obstacles.

Feature Reactions

• I found the process for choosing a conservation practice intuitive.

• The filters in VegSpec seemed comprehensive.

• From what I saw, the species in VegSpec seem well-suited to the conservation practices I’m familiar with.

• The process for calculating seeding rates seem intuitive for a planner and a customer to understand.

• The process for calculating seeding costs per pound seem intuitive for a planner and a customer to understand.

• The summary page that VegSpec generated was useful to me.

• The pdf that VegSpec generated was useful to me.

  • Open comment space to clarify on the above two questions and to ask what, if anything is not in the report that they’d like to see.

• I would find it valuable to be able to look back on my history of plant selections in VegSpec (i.e., sign in to see your last report and make tweaks)

(Final questions are open-ended.)

• What, if anything, would need to change for VegSpec to be useful for your work?

• Is there any other feedback you’d like to share?

Facilitator Questions for Focus Group Discussion

Ask respondents the following and record their answers…

• What are your impression of the location and growth requirements pages?

  • Is there anything you were expecting here that you don’t see?

  • Is there anything that seems confusing or irrelevant to you?

• What’s your impression of the goals page and the approach to setting your conservation practice?

  • Is there anything you were expecting here that you don’t see?

  • Is there anything that seems confusing or irrelevant to you?

• Do you feel you can get the information you need from the species page?

  • Is there anything you were expecting here that you don’t see?

  • Is there anything that seems confusing or irrelevant to you?

  • Are there other filters you would want on this screen when comparing species?

• What’s your impression of the seeding mix page and the approach to calculating seeding rates and costs?

  • Is there anything you were expecting here that you don’t see?

  • Is there anything that seems confusing or irrelevant to you?

• What’s your impression of the report page?

  • Is there anything you were expecting here that you don’t see?

  • Is there anything that seems confusing or irrelevant to you?

  • How best could an export feature (saving a pdf from the report) support you in your work?

• How do the factors in VegSpec (such as precipitation, cold hardiness, and soil depth) compare to the factors you currently use in your planning process?

• What’s your overall impression of VegSpec?

• Is there anything else you’d like us to know?

Log your takeaways and observations…

• How easily was the respondent able to make their way through the tool during testing?

• What challenges, if any, did they have?

• How did the session(s) go? Is there any other support that you would like to have had? (identify challenges or barriers)