Skip to content

Commit

Permalink
Merge pull request #10 from baidu/on-dev
Browse files Browse the repository at this point in the history
merge 1.6.0 to trunk
  • Loading branch information
mythmgn authored Dec 29, 2017
2 parents 81a918c + 2a28619 commit b7873f0
Show file tree
Hide file tree
Showing 9 changed files with 276 additions and 45 deletions.
93 changes: 54 additions & 39 deletions ChangeLog
Original file line number Diff line number Diff line change
@@ -1,43 +1,58 @@
Version 1.6.0 - Under development, not released yet. starting from 2017.9.6 ~
* [Enhancement] cup.mail enhanced for analyzing cc and bcc lists.

Version 1.5.6 - starting from 2017.3.1 ~ 2017.9.5
* [Enhancement] async enhancement for stability
* [New] CycleIDGenerator for generating universally unique_id (ip, port
encoded as part of the id)
* [Enhancement] cup.net.async exits more quickly than before
* [Bug] cup.net.async - Fix CPU-utilization too high bug
* [Bug] cup.net.async - Fix getting-peerinfo bug
* [Bug] cup.res.linux - Kernel version was returned with a tuple
('2', '6', '32') which should be (2, 6, 32)

Version: 1.5.5 - staring from 11.18 ~ 2017.3.1
* [Enhancement] debug method for executor
* [async] CNeedAckMsg & retry mechnism added. CAckMsg added

Version: 1.5.4 - Starting from 2016.9 ~ 2016.11.11
* [Enhancement] generator supports staring point
* [Enhancement] catch exception socket.gaierror when it encounters network
instability
* [Bug] Set up splitter other thant colon in a Configure2Dict with blanks and comments
* [Async] Support automatic msg retry
* [Async] Support ack msg
Version 1.6.0 - starting from 2017.9.6 ~ 2017.12.29
* [New] MsgBroker - Add a broker for handling system failures
* [New] cup.
* [Bug] Linux Resource Query Bug - related to data columuns
* [Bug] cup.net.async - socket cannot be got.
* [Enhancement] cup.net.async - CUP utilization enhancement

Version: 1.5.1
* [New] cup.log - add xxx_if
* [New] cup.thirdp - replace MySQLdb with pymysql.
Use can still use "from cup.thirdp import MySQLdb"
* [New] cup.util.conf - bug fix

Version: 1.5.0
* [New] cup.jenkinslib - add jenkins lib with which you can operate on jenkins jobs
* [New] cup.log.parse - parse string line logged by cup.log.XXX
* [New] cup.unittest.assert_startswith
* [New] cup.oper - add contains_file which searchs a file and return its existence
* [Bug] cup.util.conf - fix "key comparation order" bug for Configure2Dict
* [Bug] cup.util.conf - fix HdfsXmlConf "eletemnt without value. e.g. <value/>
Version 1.5.6 - starting from 2017.3.1 ~ 2017.9.5
* [Enhancement] async enhancement for stability
* [New] CycleIDGenerator for generating universally unique_id (ip, port
encoded as part of the id)
* [Enhancement] cup.net.async exits more quickly than before
* [Bug] cup.net.async - Fix CPU-utilization too high bug
* [Bug] cup.net.async - Fix getting-peerinfo bug
* [Bug] cup.res.linux - Kernel version was returned with a tuple
('2', '6', '32') which should be (2, 6, 32)

Version: 1.4.2.0
Version: 1.5.5 - staring from 11.18 ~ 2017.3.1
* [Enhancement] debug method for executor
* [async] CNeedAckMsg & retry mechnism added. CAckMsg added

Version: 1.5.4 - Starting from 2016.9 ~ 2016.11.11
* [Enhancement] generator supports staring point
* [Enhancement] catch exception socket.gaierror when it encounters network
instability
* [Bug] Set up splitter other thant colon in a Configure2Dict with blanks and comments
* [Async] Support automatic msg retry
* [Async] Support ack msg

Version: 1.5.3 - Starting from 2016.6 to 2016.8
* [New] cup.util.conf - support $include "conf_file" syntax [write/read]
* [New] port free check. Listened port probe.
* [Enhancement] cup.net.async - enhance network write/read speed.
Example provided
* [Improvement] - Improve cup.log performance
* [Enhancement] cup.util.threadpool, callback function will receive
Exception object (param result) if it encounters error.

Version: 1.5.1 && 1.5.2
* [New] cup.log - add xxx_if
* [New] cup.thirdp - replace MySQLdb with pymysql.
* from cup.thirdp import pymysql
* import MySQLdb
* [New] cup.util.generator - get_random_str
* [Bug] cup.util.conf - bug fix

Version: 1.5.0
* [New] cup.jenkinslib - add jenkins lib with which you can operate on jenkins jobs
* [New] cup.log.parse - parse string line logged by cup.log.XXX
* [New] cup.unittest.assert_startswith
* [New] cup.oper - add contains_file which searchs a file and return its existence
* [Bug] cup.util.conf - fix "key comparation order" bug for Configure2Dict
* [Bug] cup.util.conf - fix HdfsXmlConf "eletemnt without value. e.g. <value/>

Version: 1.4.2
* [Bug] oper.is_proc_exist. fix a bug while check proc exist
* [New] Add cup.services.executor. Exec and delay_exec service
* [New] Add cup.services.buffers. Buffer releated feature. For easing
Expand All @@ -51,7 +66,7 @@ Version: 1.4.2.0
* [New] cup.conf - add HdfsConf support
* [New] cup.shell.get_pid - get process id by process_path and grep_string

Version: 1.4.1.0 - [not existent on github.com]
Version: 1.4.1
* [New] assert_boundary, assert_local_file_eq 2015/4/19
* [New] Log level [warnning error fatal] splitted into file.log.wf
* [New] Hdfs
Expand Down
13 changes: 13 additions & 0 deletions LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Copyright [CUP]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
7 changes: 7 additions & 0 deletions cup/ChangeLog
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
Version 1.6.0 - starting from 2017.9.6 ~ 2017.12.29
* [New] MsgBroker - Add a broker for handling system failures
* [New] cup.
* [Bug] Linux Resource Query Bug - related to data columuns
* [Bug] cup.net.async - socket cannot be got.
* [Enhancement] cup.net.async - CUP utilization enhancement

Version 1.5.6 - starting from 2017.3.1 ~ 2017.9.5
* [Enhancement] async enhancement for stability
* [New] CycleIDGenerator for generating universally unique_id (ip, port
Expand Down
28 changes: 24 additions & 4 deletions cup/decorators.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@
:descrition:
decorators related module
"""

from datetime import datetime as datetime_in
import platform
import os
import time
import platform
import threading
from functools import wraps
from datetime import datetime as datetime_in

import cup

__all__ = ['Singleton', 'needlinux', 'TraceUsedTime']
__all__ = ['Singleton', 'needlinux', 'TraceUsedTime', 'needlinux', 'needposix']


class Singleton(object): # pylint: disable=R0903
Expand Down Expand Up @@ -94,6 +94,26 @@ def your_func():
return function


def needposix(function):
"""
only support posix
:platform:
Posix compatible
example
::
from cup import decorators
@decorators.needposix
def your_func():
pass
"""
if os.name != 'posix':
raise cup.err.DecoratorException(
'The system is not posix-based'
)
return function

# pylint:disable=R0903
class TraceUsedTime(object):
"""
Expand Down
20 changes: 19 additions & 1 deletion cup/err.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
__all__ = [
'BaseCupException', 'DecoratorException', 'LoggerException',
'ResException', 'NoSuchProcess', 'AccessDenied', 'NetException',
'AsyncMsgError', 'ThreadTermException'
'AsyncMsgError', 'ThreadTermException', 'LockFileError'
]


Expand Down Expand Up @@ -136,4 +136,22 @@ def __init__(self, msg=''):
msg = 'Not initialized: %s' % msg
BaseCupException.__init__(self, msg)


class LockFileError(BaseCupException):
"""
LockFileError
"""
def __init__(self, msg=''):
msg = 'LockFileError: %s' % msg
BaseCupException.__init__(self, msg)


class ExpectFailure(BaseCupException):
"""
Expect failure for cup.unittest
"""
def __init__(self, expect, got):
msg = 'expect failure, expect {0}, got {1}'.format(expect, got)
BaseCupException.__init__(self, msg)

# vi:set tw=0 ts=4 sw=4 nowrap fdm=indent
124 changes: 124 additions & 0 deletions cup/exfile.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*
# #############################################################
#
# Copyright (c) Baidu.com, Inc. All Rights Reserved
#
# #############################################################
"""
:authors:
Guannan Ma maguannan @mythmgn
:description:
file related functions
"""
import os
import sys
import fcntl

from cup import err
from cup import decorators

__all__ = [
'LockFile', 'FILELOCK_SHARED', 'FILELOCK_EXCLUSIVE',
'FILELOCK_NONBLOCKING', 'FILELOCK_UNLOCK'
]


FILELOCK_EXCLUSIVE = fcntl.LOCK_EX
FILELOCK_SHARED = fcntl.LOCK_SH
FILELOCK_NONBLOCKING = fcntl.LOCK_NB
FILELOCK_UNLOCK = fcntl.LOCK_UN


class LockFile(object):
"""
lock file class
"""

def __init__(self, fpath, locktype=FILELOCK_EXCLUSIVE):
"""
exclusive lockfile, by default.
Notice that the file CANNOT exist before you intialize a LockFile obj.
Otherwise, it will raise cup.err.LockFileError
:raise:
cup.err.LockFileError if we encounter errors
"""
self._fpath = fpath
self._locktype = locktype
self._fhandle = None
try:
# if FILELOCK_EXCLUSIVE == locktype:
# self._fhandle = os.open(
# self._fpath, os.O_CREAT|os.O_EXCL|os.O_RDWR
# )
# else:
self._fhandle = os.open(
self._fpath, os.O_CREAT | os.O_RDWR
)
except IOError as error:
raise err.LockFileError(error)
except OSError as error:
raise err.LockFileError(error)
except Exception as error:
raise err.LockFileError(
'catch unkown error type:{0}'.format(error)
)

def __del__(self):
"""del the instance"""
try:
if self._fhandle is not None:
os.close(self._fhandle)
# pylint: disable=W0703
except Exception as error:
sys.stderr.write('failed to close lockfile:{0}, msg:{1}'.format(
self._fpath, error)
)
sys.stderr.flush()

@decorators.needposix
def lock(self, blocking=True):
"""
lock the file
:return:
return False if locking fails
:raise:
raise cup.err.LockFileError if blocking is False and
the lock action failed
"""
flags = 0x1
if FILELOCK_SHARED == self._locktype:
flags = FILELOCK_SHARED
elif FILELOCK_EXCLUSIVE == self._locktype:
flags = FILELOCK_EXCLUSIVE
else:
raise err.LockFileError('does not support this lock type')
if not blocking:
flags |= FILELOCK_NONBLOCKING
ret = None
try:
ret = fcntl.flock(self._fhandle, flags)
except IOError as error:
raise err.LockFileError(error)
except Exception as error:
raise err.LockFileError(error)
return ret

def unlock(self):
"""unlock the locked file"""
try:
fcntl.flock(self._fhandle, FILELOCK_UNLOCK)
except Exception as error:
raise err.LockFileError(error)


# TODO tempfile
# class TempFile(object):
# """
# tempfile
# """
# vi:set tw=0 ts=4 sw=4 nowrap fdm=indent
13 changes: 13 additions & 0 deletions cup/unittest.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import logging

import cup
from cup import err

__all__ = [
'assert_true',
Expand All @@ -32,6 +33,7 @@
'assert_ne',
'CUTCase',
'CCaseExecutor',
'expect_raise'
]


Expand Down Expand Up @@ -460,4 +462,15 @@ def runcase(cls, case):
print '======== Passed ========'
print '========================'


def expect_raise(function, exception, *argc, **kwargs):
"""expect raise exception"""
try:
function(*argc, **kwargs)
except exception:
pass
else:
raise err.ExpectFailure(exception, None)


# vi:set tw=0 ts=4 sw=4 nowrap fdm=indent
2 changes: 1 addition & 1 deletion cup/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
Guannan Ma maguannan @mythmgn
"""

VERSION = '1.5.6'
VERSION = '1.6.0'
AUTHOR = 'nfs-qa'

# vi:set tw=0 ts=4 sw=4 nowrap fdm=indent
Loading

0 comments on commit b7873f0

Please sign in to comment.