This playbook provides a comprehensive approach to implementing AI-powered tagging systems for organisations that manage grants, applications, and program data. It is designed for:
This document guides you through the entire process of implementing an automated tagging system:
NGO – Non-Governmental Organisation
NLP – Natural Language Processing
LLM – Large Language Model
MeSH – Medical Subject Headings
POC – Proof of Concept
MVP – Minimum Viable Product
CI/CD – Continuous Integration/Continuous Deployment
Organisations managing grants and programs face significant challenges with manual data categorisation. Staff members often spend hours categorising applications and reports, a process that is both time-consuming and prone to inconsistency. Different team members may apply tags differently, creating data inconsistencies that compound over time and make it difficult to maintain a reliable database.
These inadequate tagging practices make it hard to generate accurate impact reports, as the underlying data lacks the metadata needed for meaningful analysis. Valuable patterns and trends remain hidden in unstructured data, preventing organisations from gaining insights that could inform strategic decisions. As data volumes grow, manual processes become increasingly unmanageable, creating a scalability bottleneck that threatens to overwhelm even well-resourced teams.
An AI-powered tagging system delivers substantial benefits that transform how NGOs work with their data. Organisations experience a significant reduction in manual tagging effort, freeing staff to focus on higher-value activities. The system standardises categorisation across the organisation, ensuring that everyone applies the same criteria consistently.
With properly tagged data, teams can generate more accurate and comprehensive impact reports that truly reflect the organization’s work. The structured data reveals patterns and trends that were previously invisible, enabling leaders to make more informed strategic decisions. Perhaps most importantly, the system handles growing data volumes without requiring proportional increases in effort, providing a sustainable path forward as the organisation scales.
The discovery phase begins with comprehensive user research to understand how different stakeholders interact with grant data. We conduct structured interviews across three key groups: grant managers who spent significant time manually categorising applications, data analysts who rely on these tags to generate meaningful reports, and leadership who needs better insights to guide strategic decisions. Through these conversations, we uncover critical requirements that shapes the entire project.
For example, during a discovery phase with a major UK grant funder, we conducted user research interviews with 11 members across different teams including Data, Policy, Strategy & Innovation, Investment, Grants Programme, and Investment Management Service. These interviews revealed several critical insights: staff members spent significant time manually categorising applications with inconsistent results across teams; different data requests required different parameters (geographic specifics, date ranges, multiple combined tags); and there was a strong need for the tagging system to support the organisation’s long-term strategy and enable compelling storytelling about impact. The research also uncovered that while reducing manual tagging time was the main presenting problem, accuracy and trust in the system were equally critical—users needed confidence that the vast majority of projects would be accurately tagged so human checking wouldn’t be excessive. Through a process-mapping session, we documented the current workflow used by the data team in responding to requests, which helped us understand exactly where AI could add the most value and how it would need to integrate with existing processes.
With user needs established, we turn our attention to the technical landscape. The discovery phase involves a thorough assessment of the organisation’s existing data infrastructure, examining the formats and volumes of grant data they process, the structure of their current tagging taxonomies, and the systems we will need to integrate with. Security and compliance requirements are also carefully documented to ensure any solution will meet organisational standards.
For the GSMA project, we needed to understand their document collection to create a specialized search system. We worked with over 200 technical specification documents plus about 7,000 additional reports and papers—roughly 100,000 pages total. These documents covered different telecom topics (like eSIM, Security, and IoT) and included specialized industry terminology.
The main technical challenges included: accessing documents (some required automated web scraping due to website protections), processing large volumes efficiently (we needed to generate hundreds of thousands of training examples while managing memory and API costs), and ensuring quality (we used filtering methods to maintain high-quality training data). Since the documents were publicly available, security was straightforward, though we needed to upload final datasets to both public and internal platforms.
Before committing to a full implementation, we conduct feasibility testing to validate the proposed approach. Using small samples of actual grant data, we evaluate baseline tagging accuracy across different technical approaches, we measure processing speeds to ensure the solution can handle typical data volumes, and assess the complexity of integrating with existing systems.
Throughout the discovery and proof of concept phases, a comprehensive ethical review process should be integrated to ensure responsible AI development. The process involves conducting workshops with diverse stakeholders across the organization to ensure multiple perspectives inform the ethical assessment of potential misuses and stress cases. The review emphasizes a sociotechnical approach that considers both the AI product design and surrounding business processes, ensuring sufficient human expert involvement throughout deployment. Regular ethics reviews and algorithmic audits should be scheduled at key project junctures, with an AI working group taking ongoing responsibility for ethical governance as the system evolves from proof of concept through to production.
The discovery phase concludes with clear recommendations for moving forward. Organizations should focus initially on few specific tagging dimensions that will deliver the highest business impact. Rather than attempting to automate everything at once, a phased approach is recommended that starts with proof of concept development.
For example, in our work with the Foundation for Innovative New Diagnostics (FIND), the discovery phase revealed that they were working with Target Product Profile (TPP) documents—PDFs containing semi-structured data in a mix of tables and free text, with up to 66 different characteristics to extract per document. The critical insight from these initial discussions was the extreme variability in document structure: some TPPs had well-formed tables while others presented data in inconsistent formats that would be difficult to parse systematically.
Based on these findings, we recommended a hybrid extraction approach that could adapt to this variability rather than relying on a single method. The key recommendation was to build a system that could intelligently route documents to the most appropriate extraction method based on their structure—using simpler, faster rule-based approaches where possible and reserving LLM-based extraction for documents that truly required it.
Rather than attempting to solve all tagging challenges at once, a strategic approach focuses the proof of concept on the highest-priority needs. This means building a complete end-to-end pipeline that can process grant applications and output the needed metadata, but initially handling only few specific tagging dimensions where the business impact will be most immediately felt.
To ensure the system is practical in real-world use, it is better to include an interface that will allow grant managers to review AI-generated tags and make corrections when needed. This human-in-the-loop approach is essential for building trust in the system while maintaining high data quality standards.
For one grant-funder client, we built a working tagging system during the proof of concept phase. The system took grant applications and automatically categorized them in two ways: identifying religious buildings (like churches and mosques) and understanding how young people were involved in projects (whether they were just attending, actively participating, or taking leadership roles). We created a simple web tool where staff could enter project descriptions and instantly see the AI’s suggestions with confidence scores. Staff could then quickly review and correct any mistakes. The system achieved 91% accuracy and turned what used to take weeks of manual work into just seconds, while still keeping human experts involved to ensure quality.
Throughout the POC development process, close contact is maintained with the client’s team to ensure the solution evolves in the right direction. Regular demonstration sessions are held where the team walks through the latest version of the system, gathering immediate reactions and noting any concerns or suggestions. These sessions are complemented by hands-on usability sessions with the grant managers who will ultimately be using the review interface day-to-day, allowing observation of how they naturally interact with the tool and identification of friction points that weren’t anticipated.
At Wellcome Trust, user research throughout the project shaped how we developed the location extraction solution. Based on this feedback, we iteratively developed a machine learning solution that could identify and extract location mentions from unstructured text at scale. User testing shaped our approach in key ways:
- We iterated on the models to handle various ways locations were referenced—from explicit city names to subtle geographic mentions in research contexts
- We prioritized extracting the location mentions most valuable for their specific analysis needs rather than all possible locations
- We built an automated flagging system for geographic mentions that seemed unlikely (like “Paris” in a UK-focused grant) so staff could verify them rather than assuming all extractions were correct
This continuous feedback loop ultimately delivered a 35% increase in available location dataand enabled Wellcome to answer the critical question: “who are the ultimate beneficiaries of research funding?”—insights previously impossible to obtain through manual methods at scale.
Feedback gathered drives concrete improvements to the system. Considerable effort goes into refining prompts to handle edge cases more gracefully—those unusual applications that don’t fit neatly into standard categories but that grant managers encounter regularly enough to matter. Recognising that not all AI predictions are equally certain, confidence scores are added to each tag, giving grant managers a way to prioritise which applications most need their careful review rather than forcing them to check everything.
When preparing the solution for production deployment, we focus on creating a comprehensive package that integrates seamlessly into the client’s existing infrastructure while remaining maintainable and scalable. The core of the system is built as a containerised API service, allowing the technical team to deploy it within their own environment with full control over security and data governance. This architectural choice gives them the flexibility to run the service on their preferred cloud platform while maintaining compliance with their data protection requirements.
Alongside the API service, we develop a streamlined web interface specifically designed for the grant managers who will be reviewing and correcting AI-generated tags. This interface prioritises simplicity and efficiency, presenting only the information needed for quick decision-making while maintaining a clear audit trail of all human corrections. We also provide extensive documentation that covers every component of the system, from API endpoints and data formats to integration points with the client’s existing grant management platform, ensuring their technical team can maintain and extend the system independently.
Recognising that successful adoption depends on proper training, we create role-specific training materials for different user groups. Grant managers receive practical guides focused on using the review interface effectively, while technical staff receive detailed documentation on system architecture and maintenance procedures. Leadership receives materials explaining the business value and how to interpret system metrics and reports.
Before releasing the system into a live environment, a rigorous testing program is conducted to validate every aspect of the solution. Performance testing is particularly critical—the system must be able to handle actual volumes of grant applications during peak periods without degradation in speed or accuracy. Using production-scale data sets, realistic workloads are simulated and any bottlenecks that might impact user experience are identified.
Security assessments form another crucial pillar of the quality assurance process. Given the sensitive nature of grant application data, close collaboration with the client’s security team verifies that all data protection measures meet stringent requirements. This includes evaluating encryption protocols, access controls, and data retention policies to ensure complete compliance with both internal policies and external regulations like GDPR.
Integration testing proves invaluable in catching issues that wouldn’t appear in isolated component testing. The tagging system’s interaction with the client’s existing grant management platform is systematically verified, ensuring that data flows correctly in both directions and that the new system doesn’t disrupt established workflows.
Finally, user acceptance testing brings the client’s actual grant managers into the process, allowing them to work with the system in realistic scenarios and provide final feedback before go-live.
To support ongoing development and maintenance, we establish a robust continuous integration and continuous deployment pipeline that automates many aspects of system updates and improvements. This infrastructure enables the client’s team to deploy updates confidently, knowing that automated processes will catch potential issues before they reach production. The pipeline includes automated deployment processes that reduce the manual effort and human error associated with releasing new versions.
A critical component of the CI/CD infrastructure is the versioning system we implement for both machine learning models and the prompts that guide the AI’s tagging decisions. This versioning allows the organization to track exactly which model version produced each set of tags, making it possible to understand and address any performance changes over time. The system also provides the ability to roll back to previous versions if a new model doesn’t perform as expected.
We build in comprehensive monitoring of system performance, tracking not just whether the system is running but how well it is performing against key metrics. Scheduled retraining workflows ensure that the models can be updated regularly with new data, allowing the system to adapt to evolving patterns in grant applications and maintain high accuracy over time without requiring manual intervention.
Establishing a comprehensive performance monitoring framework is essential to ensuring the system’s ongoing success and identifying opportunities for improvement. At the heart of this monitoring is tracking tagging accuracy by comparing the AI’s initial classifications against corrections made by human reviewers. This metric provides immediate insight into which categories or types of applications the system handles well and where additional refinement might be needed.
Beyond accuracy, processing times and system loads are monitored to ensure the solution remains responsive even as usage grows. These operational metrics help the technical team anticipate when additional resources might be needed and identify any performance degradation before it impacts users. Usage patterns and adoption rates are also tracked across different teams and user groups, providing visibility into how well the system is being integrated into daily workflows and highlighting any groups that might benefit from additional training or support.
Finally, detailed tracking of error rates and types is implemented, categorising issues to distinguish between technical failures, edge cases that require prompt refinement, and legitimate ambiguities that warrant human judgment. This granular error analysis proves invaluable for prioritising improvements and demonstrating to stakeholders that the system is continuously learning and improving based on real-world usage.
Once deployed, the tagging system becomes an integral part of daily operations at organizations. As new grant applications arrive, they are automatically processed through the AI pipeline, generating structured tags that would previously require hours of manual categorization. Grant managers adapt their workflow to focus their attention where it matters most. In one deployment, the tagging tool achieved 91% accuracy for top-level categorization, which meant staff could prioritize reviewing applications where the AI assigned lower confidence scores rather than checking every single application. This strategic allocation of human oversight means that staff time is spent on edge cases—like distinguishing whether young people were truly leading a project versus simply participating—while straightforward categorizations proceed automatically.
For the reporting teams, the impact is transformative. At Wellcome Trust, the implementation of MeSH taxonomy tagging across over 130,000 grant applications meant that questions about funding patterns that once took weeks of manual data wrangling could now be accomplished in minutes through simple database queries. The 35% increase in available location data enabled Wellcome to answer the critical question: “who are the ultimate beneficiaries of research funding?”—insights previously impossible to obtain through manual methods at scale. In another deployment, structured tagging of places of worship entries and systematic classification of youth involvement enabled the funder to respond rapidly to Freedom of Information requests and analyze funding trends over time, replacing what had been ineffective keyword searches with reliable, contextual understanding of their grant portfolio.
The first few months after launch reveal opportunities for refinement that aren’t apparent during development. Certain categories will most likely show lower accuracy than others, prompting revisiting and fine-tuning of the prompts guiding the AI’s decision-making for those specific cases. These adjustments are informed by patterns in the corrections grant managers make, creating a valuable feedback loop between the system’s users and its continuous improvement.
Technology alone doesn’t transform how organisations work—people do. Successfully implementing a tagging system requires a thoughtful approach to organisational change management that goes well beyond simply deploying software. Tailored training sessions are conducted for different user groups, recognizing that grant managers need different skills and knowledge than technical administrators or reporting analysts. These sessions aren’t just about teaching people to use a new interface; they’re about helping them understand how AI-assisted tagging fits into their broader responsibilities and how to exercise judgment when reviewing automated suggestions.
Champions emerge naturally within organisations—early adopters who see the system’s value and enthusiastically promote it to their colleagues. These champions prove invaluable in driving adoption, as peer-to-peer advocacy carries more weight than top-down mandates. Supporting these champions with talking points, demonstration materials, and direct access to the technical team when questions arise strengthens adoption. Regular check-ins with both champions and skeptics allow concerns to be addressed proactively, gathering feedback that shapes ongoing improvements and helping reluctant users see how the system can make their work easier rather than more complicated.
Working with NGOs to implement AI-powered tagging systems has revealed several important principles that can make the difference between success and failure. These lessons emerged not from theory, but from real-world implementation challenges and the solutions we developed alongside our clients.
The first and perhaps most crucial lesson is to resist the temptation to solve everything at once. When organisations first see the potential of automated tagging, they naturally want to apply it across every dimension of their data immediately. However, we’ve consistently found that starting with just one or two high-value tagging dimensions allows teams to build confidence, refine their approach, and demonstrate value before expanding. This focused beginning creates a foundation for sustainable growth rather than overwhelming users with too much change too quickly.
Equally important is maintaining human oversight throughout the process. The most successful implementations we’ve seen aren’t those that eliminate human involvement entirely, but rather those that position AI as a powerful assistant to human judgment. By keeping experienced staff in the loop to review edge cases and validate automated decisions, organizations build trust in the system while ensuring that nuanced situations receive the attention they deserve. This human-AI partnership typically delivers better results than either approach could achieve alone.
We’ve also learned that technical sophistication doesn’t always equal better outcomes. While it’s tempting to reach for the most advanced AI models available, sometimes simpler approaches—whether rule-based systems for well-defined cases or more straightforward models for specific tagging needs—actually outperform complex solutions. The key is matching the technical approach to the specific problem at hand rather than applying a one-size-fits-all solution.
The foundation of any successful tagging system is a clear, well-defined taxonomy. Ambiguous categories or overlapping definitions will undermine even the most sophisticated AI, while crisp, unambiguous tagging categories dramatically improve system performance and user confidence. Investing time upfront to clarify what each tag means and how categories relate to one another pays dividends throughout the project lifecycle.
Finally, we’ve learned that the best tagging systems are never truly finished. They evolve over time based on user feedback, changing organizational needs, and new requirements that emerge as teams discover what’s possible. Building systems with iteration in mind—both technically and culturally—ensures they remain valuable assets that grow alongside the organizations they serve.
By embracing these principles, organizations can successfully implement AI-powered tagging systems that significantly reduce manual effort while improving data quality and generating insights that would be impossible to obtain through manual processes alone.
In this 45-min boost you discover how AI can help your business become more efficient, innovate and more importantly empower your team members. AI isn’t just about efficiency or automation — it’s about empowering people.
