Market Research Company in Pakistan | Pals Research

Ask anyone who has shipped a machine learning model what went wrong the first time, and you will rarely hear “the algorithm.” You will hear about the data. Blurry images tagged as the wrong product. Customer complaints sorted into “billing” when they were really about network coverage. A medical scan outlined by someone who had never seen that kind of scan before.

That is the work AI data annotation is meant to get right, and it is the work most companies underestimate.

Quick answer: AI data annotation (also called data labeling) is the process of adding meaningful tags to raw data, such as images, text, audio or video, so a machine learning model can learn from it. A person, or a person checking a machine’s suggestion, marks what is in the data: a car in a photo, the sentiment in a review, the speaker in a recording. The model then learns patterns from thousands or millions of these labeled examples. If the labels are wrong or inconsistent, the model learns the wrong thing.

At Pals Research, we came to annotation from an unusual direction. For years our teams have designed questionnaires, run fieldwork across Pakistan and coded thousands of open-ended survey responses into usable categories. Coding a verbatim answer like “the signal drops every time I’m in the basement” into a clean category is, at its core, text annotation. The discipline is the same: clear definitions, trained people, and checks on every batch. This guide shares what that experience has taught us.

Why annotation quality decides model quality

There is an old line in computing: garbage in, garbage out. In machine learning it is closer to “garbage in, confident garbage out,” because a model trained on bad labels does not hesitate. It simply repeats the mistakes at scale.

This is not a theoretical worry. In 2021, researchers from MIT and partner institutions audited the test sets of ten of the most widely used benchmark datasets in computer vision, language and audio. They estimated that at least 3.3% of labels across those datasets were wrong, and that roughly 6% of the ImageNet validation set was mislabeled. Those are datasets built and checked by some of the best-funded teams in the world. More striking, once the labels were corrected, the ranking of which model was “best” sometimes flipped. A smaller model could beat a larger one when tested on clean labels. You can browse the actual errors at labelerrors.com.

A Google Research team reached a similar conclusion from a different angle. After interviewing 53 AI practitioners working in health, conservation and finance, they found that 92% had experienced “data cascades”: problems that start with poor data and quietly compound until a model fails in the field. Their paper’s title says it plainly. Everyone wants to do the model work, not the data work.

Here is what that means in practice for a business:

  • A small label error rate becomes a big business error rate. If 5% of your training labels for fraud detection are wrong, the model does not learn “95% right.” It learns a blurred boundary, and the cost shows up as missed fraud or blocked genuine customers.
  • Inconsistency hurts as much as inaccuracy. Two annotators who each make sensible but different calls (is a half-visible pedestrian “present” or not?) teach the model two contradictory rules.
  • Fixing labels late is expensive. Re-annotating after a model has gone to production means re-training, re-testing and often re-explaining results to stakeholders.

The lesson we take from this is simple: annotation is not a data entry job. It is a measurement exercise, and it deserves the same rigor as any serious piece of research.

The main types of AI data annotation

Different models need different kinds of labels. The table below covers the work we see most often, with the techniques involved and where each one shows up in real products.

Annotation typeCommon techniquesWhere it is used
Image annotationBounding boxes, polygons, semantic and instance segmentation, keypointsRetail shelf audits, crop disease detection, quality inspection on factory lines
Video annotationFrame-by-frame tracking, event and action taggingTraffic analytics, store footfall, sports and security footage
Text annotation (NLP)Sentiment, intent, named entity recognition, topic classificationChatbots, complaint routing, social listening, document processing
Audio annotationTranscription, speaker diarization, emotion and sound-event taggingCall center analytics, voice assistants, speech recognition
LiDAR and 3D point cloud3D cuboids, point-level segmentationAutonomous vehicles, drones, mapping
LLM and RLHF dataResponse ranking, preference comparison, rubric scoring, red-teaming promptsTraining and evaluating chat assistants and generative AI

Two of these deserve a closer look because they are growing fastest.

Text and speech in local languages. Most commercial language models still perform far better in English than in Urdu, Punjabi, Sindhi or Pashto, and they struggle badly with Roman Urdu, the mix of Urdu written in Latin script that dominates WhatsApp, Facebook comments and app reviews in Pakistan. A sentence like “service bilkul bekaar hai, refund kab milega?” carries clear sentiment and intent, but only to someone who reads it natively. Annotators who understand code-switching, regional slang and sarcasm are the difference between a model that works for Pakistani customers and one that guesses.

Human feedback for generative AI. Modern assistants are refined using a technique called reinforcement learning from human feedback, or RLHF. OpenAI’s InstructGPT paper showed how human rankings of model answers could make a much smaller model preferred over a far larger one. That work depends on trained reviewers who can judge accuracy, tone and safety consistently, which is closer to qualitative research than to clicking boxes. If you have ever run a structured qualitative study with a scoring guide, you already understand the skill.

How a professional annotation project actually runs

The quality of a dataset is decided long before the first label is drawn. This is the sequence we follow, and it borrows heavily from how good quantitative research is run: define, pilot, measure, then scale.

  1. Write the guideline before anything else. Every label class gets a definition, a few clear examples and, just as important, the edge cases. What counts as “damaged packaging”? Is a dented corner enough? A good guideline reads like a codebook for open-ended survey responses: specific enough that two people reach the same answer without talking to each other.
  2. Run a pilot on a small sample. We usually label a few hundred items with several annotators working independently. The point is not output. The point is to find where people disagree, because every disagreement is a gap in the guideline.
  3. Measure agreement, not just accuracy. Inter-annotator agreement tells you whether the task is well defined. Metrics such as Cohen’s kappa correct for agreement that would happen by chance. Low agreement on a class usually means the definition needs work, not that the annotators are careless.
  4. Train against gold-standard examples. Annotators qualify on a set of items with known correct answers before touching client data. Hidden gold items keep appearing in live batches so quality is tracked every day, not just at the start.
  5. Layer the quality checks. A typical setup is annotation, then peer review, then a senior reviewer auditing a random sample of each batch. Disputed items go back to the guideline owner, and any new rule is added to the guideline so the same question is never answered twice.
  6. Use machines where they help. Pre-labeling with an existing model can speed up simple tasks considerably. Humans then correct rather than draw from scratch. The catch is that people tend to accept a machine’s suggestion too readily, so audits on pre-labeled data need to be stricter, not looser.
  7. Deliver with documentation. A finished dataset should come with the final guideline, agreement scores, class counts and notes on known limitations. Your data science team will thank you six months later.

None of this is glamorous, and that is exactly why it gets skipped. In our experience, teams that invest a week in steps 1 to 3 save several weeks of rework later.

Five mistakes we see when companies outsource annotation

Choosing on price per label alone. The cheapest vendor often wins the contract and loses the project. A label that has to be redone costs twice. Ask what the price includes: review layers, gold-standard checks, re-work policy.

Handing over a one-paragraph brief. “Tag all the vehicles” sounds clear until someone asks about motorcycles, rickshaws, reflections in shop windows and parked cars half out of frame. If the vendor does not push back with questions in the first week, be worried.

Skipping the pilot to save time. It feels faster to go straight to full volume. It almost never is. A pilot of a few hundred items surfaces most guideline problems while they are still cheap to fix.

Ignoring domain knowledge. Labeling X-rays, legal clauses or agricultural pest damage needs people who know the field, or at least specialists who review the work. For local-language text, native speakers are not optional.

Treating data security as an afterthought. Customer chats, call recordings and ID documents carry personal information. Ask where data is stored, who can access it, whether annotators work on managed devices, and how the vendor aligns with frameworks like ISO/IEC 27001 and, for European data, the GDPR. A serious partner will answer these questions in writing.

Why Pakistan, and why a research company

Pakistan has quietly become one of the more practical places to build annotation teams. The reasons are concrete.

  • A large, educated, English-capable workforce. Universities produce a steady stream of graduates comfortable with technical guidelines written in English, and the country already has a deep freelance and IT services base.
  • Useful time zones. Lahore sits between Europe and East Asia, so a team here can turn around a batch overnight for a European client or overlap with Gulf and Asian business hours.
  • Language coverage few vendors can match. Urdu, Punjabi, Sindhi, Pashto, Saraiki and Roman Urdu are spoken by hundreds of millions of people across Pakistan and the diaspora. Models that need to serve them need native annotators.
  • Cost that leaves room for quality. Competitive rates mean clients can afford the review layers and pilots that cheap vendors cut.

Why a research company rather than a pure labeling shop? Because the hard part of annotation is not the clicking. It is defining categories that mean something, training people to apply them consistently and proving the result is reliable. That is what market researchers have always done.

At Pals Research our work follows ESOMAR standards for ethics and data integrity, and the same habits carry into annotation: written protocols, supervised teams, back-checks on every batch and clear reporting. Our field teams, who already run mystery shopping audits and nationwide surveys, can also collect original data, such as shelf photos, store videos or voice samples, when a model needs data that does not exist yet. Collection and annotation under one roof means one set of quality controls from start to finish.

If you want to see how AI is already changing research work in Pakistan, our earlier piece on AI and market research in Pakistan, available on the Pals Research blog, is a good companion read. You can also browse our full list of research services.

Frequently asked questions about AI data annotation

What is the difference between data annotation and data labeling?

In everyday use they mean the same thing. Some teams use “labeling” for simple tags, such as “cat” or “dog” on an image, and “annotation” for richer markup like outlines, entity spans or timestamps. Either way, the goal is to make raw data understandable to a machine learning model.

How much annotated data does an AI model need?

It depends on the task. A simple classifier built on a pre-trained model may work well with a few thousand good examples per class. A custom object detector for a new domain may need tens of thousands. The better question is how clean and consistent the labels are, because a smaller, accurate dataset often beats a larger, noisy one.

How do you measure annotation quality?

The common measures are accuracy against gold-standard items, inter-annotator agreement (for example Cohen’s kappa or Krippendorff’s alpha), and the error rate found in random audits of each batch. A good vendor reports all three, per class, not as a single headline number.

Can AI annotate data by itself?

Partly. Models can pre-label simple, high-volume items, and that saves time. But humans are still needed to correct errors, handle edge cases, judge subjective content and check for bias. Most production pipelines today are human-in-the-loop rather than fully automated.

What is RLHF annotation?

RLHF stands for reinforcement learning from human feedback. Annotators compare or rank answers produced by a language model, judging them for accuracy, helpfulness, tone and safety. Those preferences are used to train the model to behave better. It requires careful rubrics and well-trained reviewers.

Why outsource data annotation to Pakistan?

Pakistan offers a large English-capable workforce, native coverage of Urdu, Punjabi, Roman Urdu and other regional languages, time zones that overlap with Europe, the Gulf and Asia, and competitive costs. Working with a research-led partner adds structured quality control on top of those advantages.

Does Pals Research provide data annotation services?

Yes. Pals Research supports AI teams with text, image, audio and survey data annotation, including Urdu and Roman Urdu language work, backed by the same quality-assurance processes we use in our market research projects. Contact us to discuss a pilot.

The bottom line

Models get the headlines, but data decides whether they work. The companies that win with AI over the next few years will not necessarily have the biggest models. They will have the cleanest, most carefully defined training data, labeled by people who understand both the task and the context.

If you are planning an AI project and want a partner who treats annotation as research rather than data entry, talk to us. We usually start with a small paid pilot, so you can judge our guideline, our agreement scores and our turnaround on your own data before committing to volume.

Get in touch: contact@palsresearch.pk or visit our contact page. Our office is in Lahore, and we work with clients across Pakistan and internationally.

Related reading: Market Research in Pakistan: Quantitative vs Qualitative · Top Market Research Companies in Pakistan (2026) · Quantitative Research Services


Leave a Reply

Your email address will not be published. Required fields are marked *