Document/discuss usefulness of existing OVAL 5.11.2 tests

[evgenyz]

Let's use this discussion to gather data about OVAL tests usage.

[evgenyz]

That's what we at Red Hat (https://github.com/ComplianceAsCode/content) have for RHEL 8. RHEL 9 and 10 are not much different statistically.
RHEL8 DS OVAL Breakdown.xlsx

[vanderpol]

Thanks! I'll make sure keep those tests as un-deprecated for 5.12 and included in 6.0. For your most commonly used tests, they seem to align with what we reported as well. The outliers are your lesser used tests.

[vanderpol]

@OVAL-Community/oval-board-members I would really like to hear from several more groups on this topic. Attached is the current tracking spreadsheet, which documents several LInux tests identified by DISA and Redhat to not be deprecated.
OVAL_Test_Usage_in_Public_Content_Sept_2024._Updated_DISA_RH.xlsx

[solind]

@maxullman did you implement any Slang constructs e.g., for AIX, and are they in use in content, e.g., OVAL expressions of CIS benchmarks? Jack doesn't want to deprecate anything that's actually in use anywhere.

[dodys]

For Ubuntu, we mostly use the following tests (also from ComplianceAsCode)
On Ubuntu 20.04:

On Ubuntu 22.04:

Also checked on the oval data we generate for vulnerability management and more or less those are the ones we use the most:

Nevertheless, we (I'm speaking for Ubuntu but I can also include SUSE and probably Debian on this) are interested in using linux:apparmorstatus_test, but we couldn't so far because of the version of openscap we ship in Ubuntu 20.04 and 22.04 not supporting OVAL 5.11.2.

ubuntu-oval-tests.ods

[evgenyz]

As far as I know there is no version of OpenSCAP that supports apparmorstatus_test. Because nobody wrote the probe yet.

[vanderpol]

If there is even a strong interest in a probe for future use, we'll make sure to remove it from the to be deprecated list. If you can create some sample content for apparmorstatus_test maybe it will drive some development down the road.

[dodys]

As far as I know there is no version of OpenSCAP that supports apparmorstatus_test. Because nobody wrote the probe yet.

Yep, that would be the next step.

[dodys]

If there is even a strong interest in a probe for future use, we'll make sure to remove it from the to be deprecated list. If you can create some sample content for apparmorstatus_test maybe it will drive some development down the road.

We already have some example below on more or less how it would work:
ComplianceAsCode/content@3c5b8b8#diff-b8a0e8aa9f3ce8d472740ba68c33ed62755f02a15cabdd189a4438942e375bc6

[maxullman]

Arctic Wolf has content that uses these tests:

aix fileset_test
aix no_test
asa line_test
asa version_test
cmd shellcommand_test
esx host_acceptancelevel_test
esx host_account_test
esx host_advancedsetting_test
esx host_authentication_test
esx host_busadapter_test
esx host_coredump_test
esx host_firewallexception_test
esx host_lockdown_test
esx host_module_test
esx host_ntpserver_test
esx host_portgroup_test
esx host_service_test
esx host_vib_test
esx host_vswitchpolicy_test
esx host_webserverssl_test
esx vds_portgroup_test
esx vds_test
esx version_test
esx vm_advancedsetting_test
esx vm_device_test
esx vm_harddiskdevice_test
independent environmentvariable_test
independent family_test
independent textfilecontent54_test
independent unknown_test
independent variable_test
independent xmlfilecontent_test
ios bgpneighbor_test
ios global_test
ios interface_test
ios line_test
ios routingprotocolauthintf_test
ios section_test
ios snmp_test
ios snmpcommunity_test
ios snmpgroup_test
ios snmphost_test
ios snmpuser_test
ios version55_test
iosxe bgpneighbor_test
iosxe global_test
iosxe interface_test
iosxe line_test
iosxe routingprotocolauthintf_test
iosxe section_test
iosxe snmpcommunity_test
iosxe snmpgroup_test
iosxe snmphost_test
iosxe snmpuser_test
iosxe version_test
junos show_test
linux apkinfo_test
linux apparmorstatus_test
linux dpkginfo_test
linux inetlisteningservers_test
linux partition_test
linux rpminfo_test
linux rpmverifyfile_test
linux selinuxboolean_test
linux selinuxsecuritycontext_test
linux systemdunitdependency_test
linux systemdunitproperty_test
macos authorizationdb_test
macos gatekeeper_test
macos installhistory_test
macos keychain_test
macos launchd_test
macos plist511_test
macos plist_test
macos pwpolicy512_test
macos softwareupdate_test
macos systemprofiler_test
macos systemsetup_test
nxos line_test
nxos section_test
panos config_test
solaris smf_test
solaris smfproperty_test
unix file_test
unix interface_test
unix password_test
unix process58_test
unix runlevel_test
unix shadow_test
unix symlink_test
unix sysctl_test
unix uname_test
unix xinetd_test
windows accesstoken_test
windows auditeventpolicysubcategories_test
windows cmdlet_test
windows file_test
windows lockoutpolicy_test
windows ntuser_test
windows passwordpolicy_test
windows registry_test
windows sid_sid_test
windows user_sid55_test
windows user_test
windows userright_test
windows wmi57_test

(cmd/esx/panos are from unofficial CIS schemas)

[vanderpol]

@maxullman, so to make sure I have this fully understood, the old ESX schema in 5.11.2 is obsolete and should be removed, and it should be replaced with your updated versions? If you can throw together any type of proposal and as an issue in the OVAL repository we can discuss it and likely replace the old.

[vanderpol]

Hi @vanderpol, you can find the NXOS schema files here: https://github.com/joval/jOVAL/tree/master/scap-extensions/schemas

We developed them pre-Arctic Wolf and made them available under OVAL license terms, but simply didn't ever bother putting them through the proposal process.

We also dropped the CIS shell command test in that same directory, which they'd shared with us, also under the terms of the OVAL license.

@solind

Hi @vanderpol, you can find the NXOS schema files here: https://github.com/joval/jOVAL/tree/master/scap-extensions/schemas

We developed them pre-Arctic Wolf and made them available under OVAL license terms, but simply didn't ever bother putting them through the proposal process.

We also dropped the CIS shell command test in that same directory, which they'd shared with us, also under the terms of the OVAL license.

@solind I spent about 5 seconds looking at the NX schema, and it looks like a subset of the existing Cisco IOS and Cisco IOS XE (which are a complete copy/paste of Cisco IOS unless I'm mistaken. What are your thoughts on merging all of the CiscoIOS IOS XE and IOS NX under a single schema, and keeping the ones that are used? I've always been confused why the IOS XE schema's existed at all, it's like having a new entire schema for every version of Windows that is released. I don't see that we would actually loose any functionality, just dropping piles of redundant schemas.

[solind]

Hi @vanderpol, it's not completely analogous to Windows. I know the version tests are all different, and there are differences in the commands used to collect the data on the different OSes (IOS, IOS-XE, ASA and NX-OS). Consolidating all the tests into a single schema would definitely take some work (and someone to do it), and, there would be the question of which of the tests should be supported on which platforms.

It was Cisco who originally decided there should be different schemas for IOS, IOS-XE and ASA, and we followed their lead with the NX-OS schema. We were told by Cisco that the Nexus appliances didn't run IOS or IOS-XE, that it was a new OS, and we just took that at face value.

Notwithstanding the above, I don't know the extent to which there are actually still customers using any of these schemas.

[solind]

All that to say, if you wanted to make a change like that it would make sense to do it in 6.0 (and potentially even discard them), but not in 5.12, in my opinion.

[vanderpol]

@solind, I'm in agreement that any major change would be for 6.0. I'll change my phasing of Windows schema, and say maybe this is closer to the UNIX schema, where the same logical test is being performed, but might be implemented differently on Mac vs Linux vs Solaris vs AIX, HPUX etc... From our code on Cisco IOS to IOS XE, I'm pretty confident we literally use the exact same code, and were baffled when Cisco originally did the IOS XE spec, I vaguely recall diffing the files, and the only difference was the platform name. Typos from the old IOS were still in XE etc... Just tying to clean house and figure out what makes sense going foward.

[wmunyan]

I'm not sure how up-to-date this is, or how well maintained is the coverage, but according to the CIS docs, this is what CIS-CAT supports: https://ciscat-assessor.docs.cisecurity.org/en/latest/Coverage%20Guide/#oval-capabilities

…

On Tue, Oct 8, 2024 at 7:55 AM Jack Vander Pol ***@***.***> wrote: Thanks @maxullman <https://github.com/maxullman>, any chance you would be able to share any sample content from the 'lesser used' tests like panos, ios or esx? Honestly the biggest surprise to me was your coverage of esx, I assumed that with all of the changes to Vmware over the years that spec would likely be largely obsolete, but I'm pleasantly surprised. — Reply to this email directly, view it on GitHub <#165 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7K53ACYT6EC57V37CHRDDZ2PB35AVCNFSM6AAAAABPDFWHJKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOBXHEZDGMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[vanderpol]

Thanks Bill, although our aim here is to deprecate tests that aren't in content. In SCC we support quite a few OVAL tests (way too many) that have zero real-world content. If someone at CIS could grep through their content archives and make a list of OVAL Tests in them, that would be ideal. Or if they are publicly available and I'm just out of the loop, I'll gladly grab them and them.

[vanderpol]

I'm definitely going to need a couple sets of eyes on this, as I'm not confident that this is 100% accurate. Some high level stats so far:

• Tests with zero usage: 144 (to be marked for deprecation in 5.12 and removed in 6.0)
• Tests deprecated in 5.5-5.11, but still in use in 5.11.2: 6
• Official tests actually used: 89
• Unofficial tests used: 24 (some have proposals pending, others are needed)

OVAL_Test_Usage_in_Public_and_Private_Content_Oct_2024.xlsx

[robertgendler]

The only tests used by the macOS security compliance project are the following - I see they're covered in the above xlsx sheet. But I just wanted to add in.

accountinfo_test
authorizationdb_test
family_test
file_test
launchd_test
plist511_test
systemprofiler_test
textfilecontent54_test

[vanderpol]

Closing this discussion after OVAL board meetings and the merged pull request #182