Performance Issues

The steps in this document are intended to assist in troubleshooting performance issues within RFMS, as well as offer a general checklist of items to review for optimum performance.

While we understand performance issues can be frustrating, we also know they often require a process of elimination to diagnose and resolve. Therefore, we have tried to make the following document as comprehensive as possible by including steps compiled with years of knowledge and experience from our own RFMS technicians, as well as added input from client technicians who have shared their findings with us. 

DISCLAIMER: We advise the steps below to be completed under the supervision of a qualified local technician. This document contains many troubleshooting steps, some of which may not apply to your environment. It is the responsibility of the client's technician to ensure the safety, security, and functionality of the server and workstations. RFMS, Inc. is not liable for any harm, damage, inconvenience, loss of data, or inability to work that may occur as a direct or indirect result of the completion of any of the steps below.

Please review and note the sections/steps relevant to your setup or situation. Then, if further assistance is needed, submit a request to [e-mail] and include a copy of this checklist with your notations and a description of the issue(s). 

Section 1: Confirm General System Information

Section 2: Check Basic Settings / Configurations

Section 3: General Troubleshooting (in no particular order)

Section 4: Primary Network Diagnostics

Section 5: Virtual Server Checks

Section 6: Advanced Diagnostics

1: Confirm General System Information
STEP	DESCRIPTION	ADDITIONAL NOTES	RESULTS
1-A	Ensure using the latest version of RFMS	Follow this link for update instructions: RFMS Program Update Instruction Document
1-B	Verify all servers and workstations are on currently supported Operating Systems and/or SQL Versions, and are up to date.	Follow this link for details: RFMS System Specifications Check for updates to SQL in SQL Management Studio: click Tools > Check for Updates
1-C	Version of SQL (Version = year)? Edition of SQL (Express, Standard, etc.)?	Note: To find version AND edition, connect to the instance of SQL Server then run the following query: select @@version
1-D	Size of database files: Database (mdf): Log (ldf):	If the log file exceeds 10% of the database size, the log file should be shrunk. Follow this link to process for shrinking the log file: SQL Transaction Log File Shrink Process
1-E	Amount of Total and Free memory on server
1-F	Amount of memory utilized by SQL
1-G	Amount of available hard drive space
1-H	How often are SQL database backups run? (Full and Transaction Log Backups).	WARNING: An increasing log file may indicate your database backups are not working properly.  Follow this link for our backup instructions: RFMS Backups
1-I	Does this client use Notifications?	To check if they use Notifications: In Order Entry, go to Utilities > System Options > No Reply Email Configuration Is there information in this window, and is the box checked for Enable No Reply Email for Notifications  If so, they use Notifications.
1-J	Review settings in RFMS for User to Salesperson Mapping. If multiple Users are mapped to the same Salesperson, these will need to be changed or cleared.	To check settings: In Order Entry go to: Utilities > System Options > Users Look at the Salesperson Name field. Are there multiple users with the same Salesperson?  Note: Duplicate salesperson mapping should only be done when a user has more than one computer and their RFMS user names are different on each device.
1-K	Does this client use Enterprise Manager? If so, review the audit settings to ensure changes being tracked are not excessive.	Note: Also, run the following SQL query to check the size of your AuditLog table, and purge old records, if necessary.  sp_spaceused auditlog Follow this link for more information: Enterprise Manager User Guide
1-L	Are .NET 3.5 and .NET 4.7 (or higher) installed and enabled on all servers and workstations?	Note: Make sure all available updates are installed for both.
2: Check Basic Settings / Configurations
2-A	Check SQL Browser Service is enabled and running
2-B	Check SQL Client Configuration Settings for TCP/IP and Named Pipes. Confirm whichever protocol being used in enabled.	Note: By default you can enable both and restart the SQL Services. Then narrow down to the protocol to use once you know it works with both enabled.
2-C	Ensure there are exceptions for the following ports in the firewall on the server: 1433 – TCP/UDP, 1434 – UDP (Inbound Rules)	Note: An optional test would be to temporarily disable the firewall (private and domain) to see if this resolves the issue. If it does, reenable the firewall and check the settings.
2-D	Ensure workstations are using WIRED connections, NOT wireless.	IMPORTANT: For laptops with wireless and wired capabilities, make sure the wireless option is turned off or disabled. While laptops can be plugged in with a wired connection, and show a wired connection in the Windows Toolbar, the laptop could still be using a Wireless connection.  Check Windows Settings > Network and Internet Settings to verify which is being used.
2-E	Ensure workstations are NOT going through a 100 megabit or slower connection.	Note: For optimal network performance, we recommend at least a gigabit connection to the server. Points going from the server to the workstation should have gigabit connections as well.
2-F	Ensure users have correct permissions (Security & Sharing) to the RFMS program folder and files.	Note:  Windows updates have been known to cause issues with permissions. Therefore, it may be necessary to remove and reapply permissions to resolve issues. This process should only be done when all users are out of RFMS.  For reference, we recommend users to have Full Control, so they can create, modify and remove files, such as tmp files. Alternatively, UNC paths can be used so users do not have a mapped drive access to the folder.  Follow this link for Best Practices: RFMS NTFS and Share Permissions / Best Practices
3: General Troubleshooting (in no particular order)
3-A	Perform a reset of the windows positions.	Follow this link for instructions: Reset Windows Position
3-B	If the performance issue is persistent, the 4 following tests can be helpful. Try the same process on another workstation as a different user.  Have the affected user log into a different workstation, using their credentials, and try the same process.  Log into the affected users workstation as Administrator and try the same process.  Have the affected user log directly into the server and try the same process.	If a performance issue is experienced on one workstation but not on the other workstation, or as one user but not a different user, this typically indicates either a windows local profile issue or something on the PC itself. Try creating a brand new user, either on the user's PC or on the network (depending on the client's system). Then have the user log in with this new profile to see if this resolves the performance issue.  IMPORTANT: RFMS uses the Windows logon to create the users in the system, so every setting from the program rides on the Windows profile. Therefore, when creating the new profile DO NOT copy anything from the existing profile. Also, DO NOT copy the user's System Options in RFMS. Instead, if testing is successful with the new profile, then the System Options can be set up manually, or can be copied from another user who has similar System Options and who is not experiencing the same issues.
3-C	Do workstations connect to the network through IP phones?	If so, try bypassing it and connecting the workstation directly to the network to see if this resolves the issue.  Note: We do not recommend running workstations through IP phones as this is a switch that could cause lags in network operations.  If using IP phones, make sure it is pushing out at least 1 GB speeds.
3-D	If using a Group Policy for the mapped drive to the server, is the Group Policy set to “replace” or “update?”	If set to “replace,” edit the script to use “update.” Note: beginning with Windows 8, Microsoft made a change that can cause disconnects if the Group Policy is set to “replace.”
3-E	Try setting up a local test folder on a workstation (if local network).  If using remote desktop environment, try setting up a local test folder on the remote desktop server and pointing a set of users to this.	Follow this link for instructions on setting up a local test folder: Setting Up a Local Test Folder (for Diagnostic Purposes ONLY) Important: this test folder must only be used temporarily. The user's PC should be redirected back to the normal RFMS folder and the test folder removed once testing is complete.
3-F	If utilizing TS/RDS and users are experiencing lockups or hangs, have one or two users RDP directly to the SQL Server, bypassing the TS, and see how they perform.	If users do not experience lockups, there could be an issue that needs to be addressed with the Terminal Server or with the connection between the TS and the SQL Server.
3-G	For latency and long loading times within RFMS, set-up a cliconfg alias with the full SQL server name on the user's workstation. On the workstation, open Cliconfg (SQL Server Client Network Utility) Enable the options for Named Pipes and TCP/IP Select the Alias tab Add the alias in this format: IP address or servername\instance name	Note: also make sure there is not an old alias at the workstation still trying to connect to an old server\instance.
3-H	Modify the existing cliconfg alias to use the IP address instead of the servername. Revert back to the servername if necessary. (Start > Run > cliconfg > Click on Alias tab > highlight the alias > click edit)
4: Primary Network Diagnostics
4-A	Ping the server by name and verify that the IP address reflected is the correct internal IP address of the server.
4-B	Check for updates to the NIC and Chipset Drivers on the Server.	If there are updates available, research to confirm they will not affect the server and load accordingly.
4-C	Check Server Bios power options in regards to Energy Efficiency setting to Max or High Performance.
4-D	Check Server Bios power options in regards to internal power capping from enabled to disabled.
4-E	Check for updates to the NIC and Chipset Drivers on the Workstations.	If there are available updates, research to confirm they will not affect the server and load accordingly.
4-F	Try disabling Energy Efficient Ethernet Settings
4-G	Try disabling offloading options in Network Connections on server.
4-H	Disable the IPv6 protocol on the workstation.
4-I	Confirm the anti-virus software is not performing a pre-scan of the file prior to loading.	As a temporary test, try disabling the anti-virus on the server and workstations. IMPORTANT: If using an Enterprise Security Platform make sure to test by disabling it on the servers AND workstations.
4-J	Disable the slower speed options on the NIC and/or disable Auto Negotiation on the SQL Server, TS, and Workstations.
4-K	Set Power Options to High Performance. Also set all power hibernation and sleep mode settings to off, including network adapters.
4-L	Disable TCP over NetBIOS.
5: Virtual Server Checks
5-A	If running virtual ( Hyper-V or VMware ) check virtual NIC settings.	We have had some users need to adjust the VM Switch regarding throughput limiting.
5-B	If Hyper-V environment, check into Virtual Machine Queues.	This increases overall throughput for VM's by offloading virtual network processing to the physical adapter. Note: this may be in VMware environments also.
6: Advanced Diagnostics
6-A	Use NetMon to trace the network for: Utilization % SQL Disconnects General Network issues or disconnects SQL Keepalives Communication Link Failures Etc.	If issues found, please check with your SQL Technician for resolution.
6-B	Perform database checks	At minimum, run “dbcc checkdb” (on the RFMS database) to confirm database integrity.  Review the following linked article for further database maintenance recommendations: Best Practices for Performance, Backups, and Data Integrity
6-C	Check for SQL Service Packs and Cumulative Updates.	Cumulative updates will not show up in Windows Update. Updates must be researched through the Microsoft website.
6-D	Check the “Administrative Events” section of the Event Viewer on the workstations and/or Terminal (RDS) Server.	Look for error 1002 or 1005 pertaining to RFMS. This usually indicates a SQL or TCP drop between the WS/TS and the SQL Server. Follow this link for further information:  Diagnosing SQL and Network Disconnects (1000 series errors)
6-E	Check the SQL Server logs under the Management tab in SQL Management Studio.	Look for any errors, abnormalities or repeated failure audits.
6-F	If all users hard locking at the same time, run “aba_lockinfo” query on the RFMS database in SQL during the lock.	If there is a user with a '!!' beside their name, this is the user that started the lock and all of the users with a '!' beside their name are the users that were affected by the lock.
6-G	Use PerfMon on the server during the working hours of one business day and document the following values (Averages): Logical Disk>Disk Bytes/Sec Memory>Pages/Sec Network Interface>Bytes Total/Sec Paging File>% Usage Physical Disk>%Desk Time Physical Disk>Avg. Disk Queue Length SQLServer: Buffer Manager>Buffer Cache Hit Ratio SQLServer: Buffer Manager>Page Life Expectancy SQLServer: Buffer Manager>Free Pages System>Context Switches/sec	Please check with Microsoft on optimum values. IMPORTANT: Do not to let PerfMon run through the night as this will create incorrect data.