Easy Knowledge Transition using LLM

Easy Knowledge Transition using LLM

TLDR summary

• Knowledge Transition (KT) is the process of transferring knowledge from one team to another, such as when a new System Integrator (SI) takes over support for an application.
• KT can be a time-consuming and expensive process, and it is often difficult to ensure that all of the necessary knowledge is transferred.
• Large language models (LLMs) can be used to automate some aspects of KT, such as identifying the top and edge cases of user issues and determining the steps required to resolve them.
• LLMs can also be used to validate the accuracy of documentation and identify gaps in knowledge.
• By using LLMs, KT can be made more efficient and effective, which can help to reduce the risk of missed SLAs and improve the overall customer experience.

Key Objective of KT

Knowledge Transition (KT) is the process of transferring knowledge from one party to another. In the context of application support, KT can occur when an application is transferred from one System Integrator (SI) to another. KT can also occur when knowledge is transferred between teams within an enterprise.

KT can be a complex and time-consuming process. It is important to ensure that KT is carried out effectively to minimise the impact on the application and its users.

There are a number of factors that can affect the success of KT, including:

• The level of knowledge and experience of the parties involved
• The time available for KT
• The resources available for KT
• The level of communication between the parties involved

KT can be a costly process. However, the cost of KT can be offset by the benefits of a smooth and successful transition.

The benefits of a smooth and successful KT include:

• Reduced downtime
• Improved customer satisfaction
• Reduced costs
• Increased efficiency

When planning a KT, it is important to consider the following:

• The objectives of the KT
• The scope of the KT
• The resources required
• The timeline for the KT
• The risks involved
• The contingency plans

KT can be a complex and challenging process. However, by following the steps outlined above, you can increase the chances of a successful KT.

In the case of a customer changing to a new SI, the incoming SI must quickly assimilate and take over knowledge to ensure minimal impact to support level agreements (SLA) at takeover. To achieve this objective, incoming SI deploys experienced personnel with a hope and a prayer that KT completion will ensure a bumpless transition. So a negative ROI as this effort does not translate to any change or improvement of the application.

Knowledge transfer is essential in this scenario, so let’s explore ways to reduce risk and optimise time. The following is not a comprehensive knowledge transfer plan, but rather a subset that includes the core of the technical activity.

Pressure points during KT

Suggested Approach

To achieve the objective, one approach is to:

1. Determine the top and edge cases of user issues from an ITSM support tool.
2. Pick an issue, consider all documents (including ITSM resolution notes) as information, and determine the steps required to resolve it.
3. Use Large language models (LLMs) to answer user issues. This can help to identify documentation gaps, which can then be addressed through effective knowledge transfer sessions for incoming SI personnel.
4. To validate if the steps/actions proposed by LLMs are comprehensive for resolution, update the existing documents, obtain documents not yet shared, or compose a new document to be ingested by LLM. Rerun the same questions to validate if the results are accurate.
5. Ask for the “right” documentation in the event that the answers from LLM are not accurate.

LLMs can be a very effective tool to optimize time, reduce missed SLA risk at service takeover, and provide to-the-point issue lists from a project management perspective.

There are two suggested steps for using LLMs:

1. Train an LLM:

• Use an LLM(s) (open source or OpenAI).
• Deploy the model within the corporate firewall (not with OpenAI).
• Create training data and retrain the model.

2. Use an LLM to validate user issues:

• Use an open source or OpenAI LLM.
• Use the huggingface API to access the models (makes for easy access to models).
• Read documents and other artefacts in the local repository (avoiding sending key documents outside of the enterprise firewall).
• Use the user reported issues to get answers.

This approach can enable validations in a very short time with the possibility to silo the documents by application/modules.

Here are references to some excellent articles using this approach:

• driving-large-language-model-revolution-customer-service-and-support and you may choose to check this out.
• There is another excellent approach published, that enables working with various open source LLMs to achieve the same objective. Answering-question-about-your-documents-using-langchain-and-not-openai However do remember that the focus is on validating responses for user issues. With a few tweaks to the google colab notebook, it is easy to evaluate multiple models or tweak some of the model parameters and validate for answers.

Validation

Published here are results from a sample validation of the model: declare-lab/flan-alpaca-large model on huggingface hub

For this validation:

• I used an openly available SAP Materials Management user manual from the internet. This becomes the document that I would like to query for most often reported user issues.
• I used some most common reported user queries for SAP MM from the article published here to see if the LLM can find answers as steps from this document.

Note that some of the reported issues are rearticulated to validate the document — so it may not be a one to one usage. Our intent is to check the document information, not as a chatbot to answer the issues.

User issue: Vendor is not created for Purchasing Organisation XXXX.

Validation query used: How to create vendor for purchasing organization?

We see repetitions at the end of the answer, some tweaks are required. But with an eye on validating if this information exists, we have a clear answer.

User issue: How to extend the materials for another storage location

Validation query used: How to extend the materials for another storage location?

The answer is brief but valid. We can investigate if the question when broken down can provide us the required steps.

So we try chaining the request another way:

We get the steps to create a new storage location, but see some model hallucination at the end. These are some aspects that can be looked at if we intend using this as a chatbot to answer user queries directly.

User issue: How to Create the PO for Different Currencies.

Validation query used: How to Create the PO for Different Currencies?

We see repetitions at the end of the answer, some tweaks are required. But with an eye on validating if this information exists, we have a clear answer.

User issue: While Posting The Doc . Error Occurred,1) Account Determination for Entry 1000 BSV _ _ not Possible.2) Account Determination for Entry 1000 FRL _ _ not Possible.

Validation query used: How to setup material account determination?

We got back the steps and we gave it a good rating.

User issue: Error-Tax code does not Exist while creating MIRO (SAP screen name)

Validation query used: how to create and maintain tax code?

Observations

• LLMs are still under development, and they may not be able to answer all user questions accurately. It is important to have a human expert available to review the LLM’s answers and provide additional guidance as needed.
• LLMs can be a valuable tool for KT, but they should not be used as a replacement for human expertise. LLMs can help to automate some tasks, but they cannot replace the human touch that is essential for providing excellent customer service.

Let me know your comments and suggestions.