A Small Experiment Using AI

Introduction

We had a grantmaker that wanted to use our evaluation report to extract information and show the total score per evaluator per application and then rank the applications in descending order of score for each evaluator. They wanted to see the 1st, 2nd, 3rd, etc. choices fo each evaluator across all the applications that they evaluated. We have a collection of built-in evaluation reports, but they didn't show the requested information in this format. The grantmaker was going to export the necessary raw information using our custom evaluation report, which exports information in CSV format, but they were having some difficulty figuring out how to manipulate the information from the custom report to get the results they were looking for. They contacted us, and as an experiment we used AI to take the information and generate the results that they needed.

This is a very specific case, but it highlights the power of using our custom reports in conjunction with AI tools for analysis and visualization. We thought other grantmakers might be interested in some of the things we learned from this experiment, so we wrote a simple step-by-step set of instructions of what we did and then show the results.

Step 1 - Generate the custom report

As we said a moment ago, the goal was to see the 1st, 2nd, 3rd, etc. choices for each evaluator across all the applications that they evaluated.

With this in mind, we created a custom "grant evaluation " report with the appropriate filters for the grant cycle and grant program. The selected fields included: "Name of Organization", "Project Title", "Grant Request Amount", "Evaluator Totals 1", and "Question Notes 1". We ran the custom report and exported the results to a CSV file on our desktop.

The anonymized results, with a row of administrative information deleted from the CSV file are displayed below. Because of the possible variation of evaluators for each application, and the possible length of questions and names, we use a notation of "E" for evaluator and "Q" for question in the column headers to reduce some clutter.

Step 2 - Choose and startup an AI tool

We chose to use Claude AI since we started using it for our help and support chatbot. We have some familiarity with this tool, although we are sure any other tool would work just as well. Some popular tools include:

After starting up Claude, the first prompt we entered was.

And, before clicking on the send arrow, we copied the file from our desktop and pasted it into the prompt window. We when clicked send.

Claude then waited for us to enter our prompts.

Step 3 - Extract information from the CSV file for analysis

First, we needed to just pull out the app #, total scores for each eavaluator and the evaluator names. The "*" is a programming trick that means "wildcard". It is a way to say include anything between the letter before the asterisk and the letter after the asterisk. In this case it means get every Evaluator and every Question. Our prompt follows.

The result is below.

Step 4 - Expand and pivot the columns

Next, we had to pivot from a row structure to a column structure. We wanted to get each evaluator on their own row. This could have been easily expanded by any number of evaluators. Our prompt follows.

The result is below.

Step 4 - Sort the results

Finally, we wanted to sort and order things to get our desired results. Sorting by "q1 - evaluator name" first sorts by evaluator name, and then sorting by "descending order for score" shows us the ranking of apps for each evaluator in high to low order. The prompt follows.

The results are below.

Conclusion

And, there we have it. Exporting the data from our system, and a few simple AI prompts using the data, got the results our grantmaker was looking for. If they wanted to print out the information they could have then used any of the following prompts.

This simple experiment highlighted the power of combining our exportable custom reports with external AI tools.

There is one serious caveat in all of this. Applicants have a certain expectation about the privacy of the information they provide thru our system to a grantmaker. We have a policy for data privacy and security, which we take very seriously and adhere to.

Grantmakers may have a different set of data privacy and security policies. Those policies set the expectations with applicants about how grantmakers will manage and protect their information. Those expectations are established and maintained by the individual grantmakers and we don't have any control over that.

Exporting data out via a custom report takes it out of the confines of our system and injecting it into an AI prompt potentailly risks it becoming part of the training data used by AI, which then means it may become publicly available in some way. This may or may not be a problem for applicants. Data privacy in AI is a complicated and evolving topic, but it is essential that you think through what information you may be supplying to AI, and if it gets into the public domain, will that become a problem.

In this experiment the information supplied was minimal, and judged to be okay if it got into the public domain.