mOS for HPC v0.6 Readme

What is mOS for HPC

mOS for HPC is an operating systems research project at Intel, targeting extreme scale HPC systems.  It aims to deliver a high performance computing environment with the scalability, low noise, and repeatability expected of lightweight kernels (LWK), while maintaining overall Linux compatibility that HPC applications need.  

mOS for HPC remains under development at this time. These materials are being made available to interested parties to explore, to test drive, and to provide feedback through the mailing list. Consider the quality level to be pre-alpha and as-is. It is not intended to be used in production or for business critical uses. The required knowledge level of users is expected to be expert for Linux internals and for operating system principles. Support is limited by the development team's ability to respond through the mailing list. Please note that these are the only mechanisms for interaction with Intel on this project.

What's new for v0.6?

Feature	Description	Learn More
mOS views	mOS for HPC now supports context sensitive views of various sysfs and procfs files. These views describe system resources (CPUs and memory) from the perspective of a given process.  A process can switch its view to see Linux resources or LWK resources or both. For example, in the Linux view the process sees resources that are only available to regular Linux (FWK) processes and does not include resources that are designated for LWK usage. A new tool (mosview) provides the capability to query or set the view of a given process or launch a new process in the specified view.	See Documentation/mOS-view.txt
Scheduler	The LWK CPUs are placed in a low power state when no job is running in the LWK. Also allow Linux CPUs to enter low power state by no longer recommending the setting of the boot param 'intel_idle.max_cstate=1'. Allow TSC reliability checking to occur on the Linux CPUs without introducing interrupt activity on the LWK CPUs. This allows the detection of TSC drift and the switching of the clock source to the HPET timer for both Linux and LWK timekeeping if necessary. With this change we no longer recommend the setting of the boot param 'tsc=reliable'. Disable correctable machine check interrupts and correctable machine check polling on LWK CPUs reserved by a job running on the LWK to reduce noise. When the job ends, automatically re-enable polling and interrupts and drain the machine check buffers of correctable error information that may have occurred during the running of the job. With this change we no longer recommend the setting of the boot param 'mce=ignore_ce'.   Additional kernel trace points have been added and the verbosity of existing trace points have been improved.
UTI	Improve handling of UTI requests for running on an exclusive LWK CPU. This request will now be honored in more situations. Progressively relax placement of utility threads if initial request cannot be satisfied. For example, if request was to place on the same L1 and no other CPU on that L1 was available, placement on a CPU in the same L2 will be attempted next. When a request to place a utility thread in the Linux scheduler is made, allow the thread to be run on all available Linux CPUs that match the request, not just one CPU. This allows the Linux scheduler to better balance the scheduling of threads on the limited number of resources it has available.
Debug	mOS for HPC v0.6 supports tracing and debug of the memory subsystem via the Linux ftrace facility.  This is consistent with scheduler trace, which was previously introduced. Additional scheduler trace points have been added, along with improved verbosity of existing scheduler trace points.	See Tracing mOS Code in the mOS for HPC v0.6 User Guide
yod	mOS for HPC v0.6 contains improvements to yod's NUMA aware resource reservation algorithm, providing improved scalability on nodes with very large number of NUMA domains.
Linux 4.9.107	mOS for HPC v0.6 is based on the Linux 4.9.107 kernel.
SMAP support	mOS for HPC now supports the Supervisor Mode Access Protection (SMAP) feature of the Intel Architecture.

 

Platform requirements

The development platform for mOS for HPC v0.6 has been primarily Intel(R) Xeon Phi(TM) processors 7250 with 96GiB of DRAM and 16GiB of MCDRAM and booted in SNC-4 cluster mode and flat memory mode.  Some testing of mOS for HPC v0.6 has been performed on certain models of Intel(R) Xeon(R) processors.  Your mileage may vary on other platforms and configurations in terms of functionality and performance:

• If you use the Intel(R) Xeon Phi(TM) processor 7230, then Quadrant cluster mode, Flat memory mode is recommended.
• If you want to make all of MCDRAM available to applications on Intel(R) Xeon Phi(TM) processors, you must verify that MCDRAM is hot-swap-able in the BIOS settings.  Please see the Administrator's Guide.
• The development team has observed lower performance of mOS for HPC when running in cache memory mode on Intel(R) Xeon Phi(TM) processors, which is not necessarily attributed to hardware.
• Limited testing has been performed on systems with the Intel(R) Xeon(R) processor E7-88xx, the Intel(R) Xeon(R) Platinum 8168 processor, the Intel(R) Xeon(R) Gold 6138 processor, and the Intel(R) Xeon(R) Gold 6140 processor .
• Processors outside of the x86_64 architecture designation in Linux are unsupported – the kernel code will not configure and build.

The Linux distribution used by the development team for building, installing, and testing mOS for HPC has been CentOS 7.  Other distributions have had almost no testing, and may require that you adapt the build, install instructions to your environment. 

mOS for HPC has been tested with applications using Intel(R) Parallel Studio XE 2018 Cluster Edition Update 3 for Linux*, which includes Intel MPI (2018.3.222).  The development team plans to track the Intel(R) Software Tools 2018 programs and MPICH/MPICH4 updates as they become available.  Almost no testing has been done using other compilers (e.g. gcc) or MPI runtimes (e.g. MPICH or MVAPICH or OpenMPI). 

Where to get code

The mOS for HPC source can be checked out from GitHub at https://github.com/intel/mOS.   Please see the Administrator's Guide for further instructions.

Where to report issues or ask questions

Register for the mOS for HPC mailing list at https://groups.google.com/g/mos-devel/. Please, submit feedback and follow discussions through this list.

 

^{*Other names and brands may be claimed as the property of others.}